UNCLASSIFIED 


_ AD  NUMBER _ 

AD822635 

LIMITATION  CHANGES 
TO: 

Approved  for  public  release;  distribution  is 
unlimited. 


FROM: 

Distribution  authorized  to  DoD  only; 
Administrative/Operational  Use;  OCT  1967.  Other 
requests  shall  be  referred  to  Defense  Advanced 
Research  Projects  Agency,  TIO,  Washington,  DC 
20301. 


_ AUTHORITY 

darpa  notice,  10  nov  1977 


THIS  PAGE  IS  UNCLASSIFIED 


AD822635 


FOR  OFFICIAL  USE  ONLY 


??  t 


CR-0040-1/1 

PROJECT  ECHO 


CONTRACT  DAHC 04-6 7C-0 040 


October  1967 


R.  P.  Barthol 
0.  Bridge 


SPONSORED  BY 

Advanced  Research  Projects  Agency 
Project  AGILE 
ARPA  Order  965 


general 


CORPORATION 

'•O  ,0X  JM7<  SANT*  »*"•*«*.  CALIFORNIA  .3106 


STAnaOXT  #4  UNCLASSIFIED 


D  D  C 

EDQJGEn) 

!l 


. 


Xaoh  transmittal  of  this  document  out 
Dafense  must  have  prior  approval  of 


NOV  2  0  1967 

bdD'Jrii  U'TS^ 

c 


FOR  OFFICIAL  USE  ONLY 


THIS  REPORT  HAS  BEEN  DELIMITED 
AND  CLEAREr  FOR  PUBLIC  RELEASE 
UNDER  DOD  DIRECTIVE  5200,20  AND 
NO  RESTRICTIONS  ARE  IMPOSED  UPON 
ITS  USE  AND  DISCLOSURE, 

DISTRIBUTION  STATEMENT  A 

APPROVED  FOR  PUBLIC  RELEASE; 
DISTRIBUTION  UNLIMITED, 


UNCLASSIFIED 


SUMMARY 

TASK 

ANALYSIS  OF  ALIEN  CULTURES:  VALUES  AND  POWER  STRUCTURES 

Phase  1  of  Project  ECHO  was  intended  to  develop  a  method  for  under¬ 
standing  alien  cultures  and  communicating  more  effectively  with  these 
cultures  by  being  able  to: 

1.  Identify  subcultures  within  a  population 

2.  Identify  the  value  systems  of  these  subcultures 

3.  Determine  how  far  specific  values  held  by  members  of 
the  subculture  were  self-enforced  rather  than  imposed 
by  authority 

4.  Measure  differences  in  social  perceptions  among  members 
of  a  culture 

5.  Identify  the  "power  structure"  (sources  of  influence) 
operating  in  the  culture 

METHOD 

PROJECTIVE  SURVEY:  AN  INVERTED  PUBLIC  OPINION  POLL 

The  ECHO  technique  uses  a  "projective  survey,"  reversing  the  normal 
public  opinion  polling  process:  polls  ask  the  respondent  to  assign  an 
evaluation  to  a  preselected  topic;  the  ECHO  technique  assigns  an  evalua¬ 
tion  and  asks  the  respondent  to  think  of  a  behavior  which  carries  this 
evaluation.  The  questions  used  in  the  ECHO  studies  were:  "What  is  a 
good  [bad]  thing  that  you  could  do  which  someone  would  praise  [blame]  you 
tjr  doing?  Who  is  the  someone  who  would  approve  [disapprove]?" 

DATA  BASE:  VALUES  AND  POWER  STRUCTURE 

Answers  to  the  query  "What  is  a  good  [bad]  thing?"  provided  the 
basic  data  for  the  determination  of  value  systems.  Answers  to  the  query 
"Who  would  approve  [disapprove]?"  provided  information  about  the  subjects' 
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perceived  "power  structure,"  i.e.,  the  sources  of  positive  and  negative 
reinforcement  which  control  his  behavior. 

INDIGENOUS  CLASSIFIERS 

The  classification  process,  an  integral  part  of  the  ECHO  method¬ 
ology,  is  based  on  the  hypothesis  that  indigenous  classifiers  (members 
of  the  exact  population  under  study)  are  able  to  make  fine  discriminations 
among  statements  from  their  own  subculture  and  to  understand  the  nuances 
of  the  language,  and  that  they  do  this  better  than  "experts." 

DATA  COLLECTION  AND  ANALYSIS 

Subjects  were  selected  from  the  nine  populations  being  studied  and 
were  asked  to  write  ten  anonymous  answers  to  each  of  the  ECHO  questions  on 
preprinted  and  coded  IBM  tabulating  cards. 

The  r*  ^ponse  cards  were  divided  into  logical  groups  (male-female, 
good-bad,  etc.).  Each  group  of  data  was  categorized  by  three  different 
teams  cf  classifiers,  who  divided  the  cards  into  categories  of  statements 
that  "meant  the  same,"  and  provided  brief  descriptions  of  each  category's 
contents. 

The  coded  cards  were  submitted  to  a  five-step  computer  analysis  which 
produced  value  system  and  power  structure  outputs.  Statements  represent¬ 
ing  these  value  systems  and  value  systems  of  other  groups  were  presented 
in  a  forced-choice,  paired-comparison  format  to  a  second  sample  of  subjects 
from  the  same  population.  This  was  done  as  an  estimate  of  validity — to 
see  if  ECHO-generated  data  represented  important  and  prevalent  values  in 
the  subject  culture  and  also  to  assess  the  method's  sensitivity  in  dis¬ 
tinguishing  among  similar  subcultures. 
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RESULTS 


VALIDITY  TESTS 

The  validity  „est  results  Indicated  that  the  ECHO  instrument  could 
determine  not  only  important  attitudes  within  the  subject  population  fcgt 
also  many  of  the  differences  among  various  subgroups  within  the  population. 

POWER  STRUCTURE:  INTERNALIZATION  OF  VALUES 

The  Power  Structure  Analysis  element  of  the  ECHO  method  permitted 
identification  of  the  perceived  sources  of  reinforcement  and  changes  in 
the  pattern  of  power  impinging  on  the  individual.  The  studies  indicate, 
for  instance,  that  children  in  the  third  grade  tended  to  view  their 
parents,  particularly  mother,  as  the  most  important  source  of  approval  and 
disapproval.  College  students,  on  the  other  hand,  cited  "self"  as  the  most 
important  source.  The  college  students  appeared  to  have  "internalized"  the 
value  system  of  the  culture;  the  third  graders  still  used  their  parents 
to  tell  them  what  society  thinks  is  right  or  wrong.  Sixth  grade  children 
responded  more  like  the  college  students. 

INTER-CULTURAL  DIFFERENCES 

The  Power  Structure  Analysis  is  also  capable  of  detecting  cultural 
differences  in  sources  of  reinforcement.  For  instance,  "Anglo"  univer¬ 
ity  students,  both  male  and  female,  cited  "myself"  as  source  of  approval 
far  more  often  than  Cuban  subjects  did.  Males,  in  both  cultures,  cited 
myself  more  often  than  did  their  female  counterparts;  the  degree  of 
difference  between  males  and  females  on  this  dimension  might  be  thought 
of  as  an  index  of  the  cultures'  belief  in  the  "equality"  of  the  masculine 
and  feminine  roles. 

INTRA- CULTURAL  DIFFERENCES 

A  behavior  may  be  reinforced  by  two  completely  different  sources  in 
the  same  culture,  and  the  ECHO  power  structure  analysis  can  detect  this 
fact.  For  example,  the  UCLA  students  ^aw  "getting  married"  as  a  good 
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thing  to  do.  When  asked,  "Who  would  approve?",  the  men  said  "myself"  nine 
times,  "my  parents"  one  time;  the  women  said  "my  parents"  ten  times, 
"myself"  one  time. 

DIFFERENTIATING  BETWEEN  POPULATIONS  AND  SUBPOPULATIONS 

Value  hierarchies  discovered  by  ECHO  differentiate  between  different 
populations  (e.g.,  Cubans  and  United  States  college  students)  and  between 
subpopulations  in  a  single  culture  (e.g.,  males  and  females). 

VALUE  OF  INDIGENOUS  CLASSIFIERS 

Experiments  in  which  teams  from  different  populations  classified 
data  from  a  single  population  demonstrated  the  value  of  indigenous 
classifiers;  these  individuals  are  able  to  impart  subtle  meanings  and 
differentiation  to  the  data  that  "experts"  often  overlook. 

QUANTIFYING  DIFFERENCES  IN  SOCIAL  PERCEPTION 

Differences  in  how  two  populations  view  the  same  behavior  can  be 
detected  and  quantified  by  having  a  single  batch  of  data  classified  by 
classifiers  from  both  groups. 

REDEFINING  "VALUE  DEFINITIONS" 

Meanings  assigned  by  a  group  of  indigenous  classifiers  can  be  re¬ 
defined  by  having  several  other  groups  of  classifiers,  from  the  same 
population,  classify  the  data. 

USE  OF  CHILDREN  AS  INDICATORS  OF  PARENTS’  CULTURE 

Phase  1  pilot  studies  indicate  that  children  eight  years  old  and 
older  are  capable  of  handling  the  standard  IBM  form  of  the  questionnaire. 
Research  with  children  is  preferable,  in  some  cases,  to  experiments  with 
adults:  children  are  reliable,  less  suspicious  and  more  cooperative,  and 
usually  available  over  a  long  period  of  time  (e.g.,  one  semester,  one 
school  year).  ECHO  data  collected  from  children  also  appear  to  be 
useful  as  indices  of  the  culture’s  social  control  system  and  the  value 
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system  of  the  parents'  society.  Research  should  be  undertaken  to 
discover  (1)  the  extent  to  which  data  can  be  collected  and  classified 
by  children  and  (2)  the  nature  of  the  linkage  between  children's  per¬ 
ceptions  of  their  parents'  society  and  reality. 

SENSITIVITY  TO  DYNAMIC  CHANGES 

The  ECHO  methodology  appears  to  be  capable  of  detecting  dynamic 
changes  in  the  value  structure  of  populations.  This  will  provide  a 
sensitive  instrument  for  tracing  the  internalization  of  particular 
values  and  the  socialization  process  in  a  culture  as  well  as  estimating 
the  impact  of  an  event  on  a  population. 

"POWER  STRUCTURE"  ANALYSIS 

Perceived  sources  of  positive  and  negative  reinforcement  operating 
on  a  population  (the  "power  structure")  car  be  identified  and  quantita¬ 
tively  described  by  the  current  ECHO  analysis  system. 
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INTRODUCTION 

INTERCULTURAL  COMMUNICATION  PROBLEMS 

Effective  communication  with  an  alien  culture  has  been  recognized 
historically  as  an  immensely  difficult  task.  The  difficulties  involved 
gave  rise  to  such  stereotypes  as  the  "inscrutable  Chinese"  or  the  "mys¬ 
terious  Orient"  which  implicitly  admitted  that  Westerners  simply  did  not 
understand  the  assumptions  and  values  upon  which  people  of  foreign  cultures 
acted  and  reacted.  Without  such  knowledge  and  understanding,  it  has  been 
almost  impossible  to  transmit  or  receive  messages  which  make  sense  on 
either  side  of  a  cultural  barrier. 

NEED  FOR  INTERNATIONAL  AND  INTERCULTURAL  COMMUNICATION 

However,  since  the  United  States  has  become  involved  in  many 
different  parts  of  the  world,  it  has  become  increasingly  necessary  for 
Americans  to  communicate  effectively  and  accurately,  and  to  be  able  to 
present  credible  and  persuasive  arguments  to  foreign  peoples.  At  the 
same  time,  many  allies  of  the  United  States  (and  the  United  States  itself) 
have  faced  serious  problems  of  understanding  and  communication  with  sub¬ 
cultures  and  subgroups  within  their  own  societies.  It  seemed  highly 
desirable,  therefore,  to  attempt  the  development  of  a  methodology  through 
which  a  perceptive  understanding  of  alien  value  systems,  aspirations,  and 
fears  could  be  achieved  and  intercultural  communications  improved. 

PROJECT  ECHO 

To  accomplish  this  purpose,  General  Research  Corporation  in  the 
summer  of  1966  began  the  development  of  a  technique  based  on  an  idea  of 
Professor  Alex  Bavelas  of  Stanford  University.  This  concept  avoided  the 
"previous  knowledge"  problem  inherent  in  the  normal  polling  technique 
where  people  are  asked  to  evaluate  topics  with  which  they  are  presumably 
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familiar  and  on  which  they  hold  some  opinion.  Instead,  people  were 
assigned  evaluations,  i.e.,  "good"  or  "bad,"  and  asked  to  attach  topics 
or  actions  which  corresponded  in  their  minds  with  the  evaluation.  With 
this  projective  form  of  question,  cultural  values,  aspirations,  and  fears 
could  be  evoked  from  a  population  about  which  the  investigator  might  know 
little  or  nothing. 

PROPOSED  USE  IN  THAILAND 

The  results  of  the  early  experiments  were  encouraging,  and  so, 
beginning  in  January,  1967,  ARFA  began  supporting  the  further  development 
of  the  technique  with  the  objective  of  using  it,  if  successful,  in  Thai¬ 
land.  The  problem  of  concern  to  ARPA  lay  in  the  Northeast  provinces  of 
that  country  where  there  were  signs  of  insurgency  fomented  by  Communist 
infiltrators.  It  was  thought  that  if  the  Royal  Thai  Government,  as  well 
as  the  United  States  Government,  could  gain  a  clearer  insight  into  the 
values  and  aspirations  of  the  people  in  these  remote  provinces,  more 
effective  communications  could  be  established  and  counterinsurgency  pro¬ 
grams  could  conform  more  realistically  to  the  needs  of  the  situation. 

DEVELOPMENT  OF  METHODOLOGY 

With  the  objective,  then,  of  developing  in  the  United  States  a 
viable  tool  for  use  in  Thailand,  General  Research  has  experimented  with 
the  technique,  using  subjects  from  progressively  more  difficult  or  com¬ 
plex  populations.  Beginning  with  groups  of  industrial  clerks  and  secre¬ 
taries,  the  invfeStigators  moved  on  to  university  student  groups,  primary 
school  classes,  exiled  Cubans,  and  finally  to  functionally  illiterate 
Mexicans.  With  each  successive  trial,  the  method  and  techniques  for 
taking  data,  classifying  them,  analyzing  and  validating  them  were  refined 
and  Improved;  and  it  was  demonstrated  that  language  and  cultural  barriers 
could  be  overcome  at  least  between  two  foreign,  but  still  Western,  cultures. 

PRELIMINARY  RESULTS 

While  the  research  reported  in  this  publication  is  incomplete,  it 
does  strongly  Indicate  that  a  useful  and  effective  tool  for  improving 
lntercultural  understanding  and  communications  has  been  found. 
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IMPLEMENTATION 

The  rest  of  this  report  describes  the  completed  research  in  detail 
and  discusses  some  of  the  further  research  required.  Moving  the  project 
to  Thailand  will  require  approximately  six  months'  work  in  the  United 
States  and  a  two-month  pilot  study  in  Thailand.  The  former  is  necessary 
because  the  methodology  is  not  yet  adequate  tr  obtain  the  required  infor¬ 
mation  reliably;  the  latter  is  necessary  because  we  do  not  know  the  im¬ 
pact  of  an  Eastern  culture  and  language  on  ECHO.  To  ascertain  the 
impact ,  cooperative  and  easily  available  subjects,  such  as  a  university 
class,  will  be  tested  and  the  results  will  be  validated.  When  that  is 
completed,  the  detailed  plans  will  be  made  for  working  with  Thai  popula¬ 
tions  of  interest  to  either  the  Thai  or  US  governments. 
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I.  THE  TASK 

Research  conducted  under  Phase  I  of  Project  ECHO  has  sought  to 
develop  a  methodology  for  understanding  alien  cultures  and  communicating 
more  effectively  with  these  cultures.  The  former  objective,  understanding, 
accrues  from  the  method's  ability  to: 

1.  identify  subcultures  within  a  population 

2.  identify  the  value  systems  of  these  cultures 

3.  determine  the  degree  of  internalization  of  specific  values 
of  members  of  the  subculture 

4.  quantify  differences  in  social  perceptions 

5.  identify  the  ''power  structure"  operating  in  the  culture 

The  ECHO  technique  utilizes  a  "projective  survey"  format;  the  method 
is  projective  because  the  subjects  (Ss)  are  permitted  to  define  some 
variables  which  are  usually  predetermined  by  the  investigators,  and  a 
survey;  because  groups  rather  than  individuals  are  the  object  of  concern. 

The  value  of  this  type  of  attitude  survey  and  the  theoretical  logic 
underlying  its  development  are  explicated  below. 
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II.  THEORETICAL  CONSTRUCTS  AND  ASSUMPTIONS 

Current  theories  hold  that  attitudes,  which  ECHO  attempts  to  assess, 
have  three  components:  cognitive ' (with  focus  primarily  on  the  evaluative 
Part) ,  affective,  and  behavioral.  People  tend  to  seek  consonance  among 
the  three  components,  so  that  a  given  attitude  has  internal  consistency: 
if  the  evaluation  of  an  object  is  positive,  then  the  feelings  about  and 
b shavior  toward  that  object  are  likely  also  to  be  positive.  A  person 
who  evaluates  an  object,  person,  or  concept  positively  will  be  predisposed 
to  behave  positively  toward  it;  conversely,  a  person  is  likely  to  act  in 
opposition  to  an  object,  person,  or  concept  that  he  evaluates  negatively. 

If,  on  the  other  hand,  the  system  is  out  of  balance  (e.g.,  feelings 
and  evaluation  are  positive  while  behavior  is  negative),  the  individual 
has  a  strong  tendency  to  bring  it  into  balance  by  modifying  either  the 
behavior  or  the  feelings  and  evaluation. 

Values  can  be  considered  to  be  enduring  systems  of  positive  or 
negative  evaluations.  Thus,  if  a  value  system  is  understood,  correspond¬ 
ing  behaviors  can  be  estimated;  correspondingly,  behavior  can  be  influenced 
by  modifying  values.  The  primary  ECHO  task  is  to  discover  value  systems 
by  a  method  that  is  analogous  to  survey  sampling  (polling). 

With  the  recent  advances  in  polling  techniques,  very  small  samples 
of  data  can  be  used  with  great  confidence  to  predict  public  opinion  on 
a  wide  range  of  issues.  However,  polls  are  only  applicable  to  populations 
about  which  much  is  already  known.  Asking  the  right  question  of  the 
right  people  is  difficult  even  in  a  culture  with  which  we  are  intimately 
familiar.  The  ECHO  method  obviates  this  "previous  knowledge"  problem 
by  reversing  the  polling  process.  Polls  ask  the  respondent  to  assign  an 
evaluation  to  a  preselected  topic;  the  ECHO  technique  assigns  an  evaluation 
and  asks  the  respondent  to  think  of  a  behavior  which  carries  this 
evaluation. 
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A  specific  example  of  this  kind  of  projective  question  is:  "What  is 
a  good  thing  that  you  could  do  which  someone  would  praise  you  for  doing? 
Who  is  the  someone  who  would  approve?"  The  format  of  the  question  can 
be  modified  by  varying  the  assigned  evaluation  and  "role".  The  role  in 
this  example  was  "you".  Another  role  might  be  "you  as  a  nurse"  (student, 
employee,  etc.).  Questions  can  be  cast  in  several  form  5  according  to  the 
needs  of  the  investigators. 

Answers  to  the  query  "Who  would  approve  (disapprove)?"  provide 
information  about  the  subjects'  perceived  "powerstructure" ,  i.e.,  the 
sources  of  positive  and  negative  reinforcement  which  control  his  behavior. 

The  projective  survey  technique  was  conceived  by  Professor  Alex 

1  2  3 

Bavelas  and  applied  in  a  variety  of  settings  by  Kalhorn,  Warner, 

4  5 

Havinghurst  and  Neugarten,  and  Rice.  Research  conducted  under  Project 
ECHO  has  differed  from  previous  studias  in  a  variety  of  ways.  The  current 
research,  for  example,  uses  indigenous  classifiers  rather  than  "experts1' 
to  categorize  the  raw  inputs,  and  validity  tests  have  been  introduced. 

New  forms  of  the  instrument  and  methods  of  statistical  analysis  have 
also  been  developed  during  this  period. 
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HI.  THE  ECHO  METHODOLOGY 

The  ECHO  methodology,  as  developed  by  General  Research,  is  a  nine- 
step  process  from  data  collection  to  production  of  valid  descriptions 
of  value  hierarchies  and  power  structures  in  the  test  population. 

Figure  1  represents  the  process  diagramatically. 

A  brief  summary  of  the  significant  elements  in  the  ECHO  methodology 
follows : 

1.  The  problem  is  tentatively  defined  and  an  appropriate  form 
of  two  projective  ECHO  questions  is  selected. 

2.  A  population  is  identified  and  a  sample  selected. 

3.  Subjects  are  asked  to  generate  ten  anonymous  answers  to  each 
of  the  ECHO  questions.  Answers  are  recorded  on  preprinted 
and  coded  IBM  tabulating  cards. 

4.  The  response  cards  are  divided  into  logical  groups  (male- 
female,  good-bad,  etc.)  and  each  group  of  data  is  categorized 
by  three  different  teams  of  classifiers  (Cs).  Cs  divide  the 
cards  into  categories  and  provide  brief  descriptions  of  each 
category's  contents.  They  then  rank  the  categories  on  some 
assigned  dimension  of  importance. 

5.  The  classified  cards  are  punched  with  the  two-digit  codes 
which  identify  the  classifying  team  and  the  category  descrip¬ 
tion  which  was  assigned.  Coded  cards  are  submitted  to  a  five- 
step  computer  analysis  which  produces  value  system  and  power 
structure  outputs. 

6.  Equal-length  lists  of  "values"  are  extracted  by  selecting  the 
titles  of  the  five  categories  with  the  highest  frequencies  (f) 

(i.e.,  the  largest  number  of  cards)  in  each  classification 
system. 

7‘  lists  are  Presented  in  a  forced-choice,  paired-comparison 

format  to  a  second  sample  of  subjects  from  the  same  population. 
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Each  list  is  paired  with  one  of  the  others,  or  with  a  list  from 
some  other  population;  Jte  are  asked  to  select  the  set  of  values 
in  each  pair  which  is  "most  important  to  them".  The  hypothesis 
is  that  the  Ss  will  select  (significantly  more  than  chance)  the 
list  which  came  from  their  own  population  and  will  reject 
(significantly  more  than  chance)  lists  which  did  not  come 
from  their  population. 

8.  Those  lists  which  were  selected  as  hypothesized  are  assumed 
to  come  from  valid  represente.tions  of  a  portion  of  the 
population's  value  hierarchy. 

9.  Concurrently  with  the  above,  other  data  are  analyzed  and  the 
results  combined  with  (8) . 

The  above  apercu  constitutes  only  a  cursory  outline  of  the  ECHO 
methodology.  A  detailed  description  and  justification  of  each  variable 
in  the  process  (20  cards,  3  classifications,  5  most  frequently  mentioned 
categories,  etc.)  is  given  in  Sec.  V  through  VII.  The  specific  way  in 
which  the  methodology  was  applied  in  each  of  the  populations  that  have 
bAen  studied  is  described  in  Appendix  I  (see  in  particular  Appendix  I, 

Sec.  J,  for  the  most  recent  fom).  A  generalized  statement  of  the 
methodology  is  given  in  Appendix  IV. 

The  methodology  described  above  is  the  product  of  a  large  number 
of  individual  experiments.  The  contribution  that  each  experiment  made 
to  a  particular  element  of  the  methodology  is  described  below;  the 
chronological  sequence  of  events,  hypotheses,  and  experiments,  are  reported 
in  Appendix  I. 
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IV.  SUBJECT  POPULATIONS  USED  IN  PHASE  I 

Eleven  different  populations  participated  in  the  Phase  1  research. 
These  groups  included:  freshmen,  seniors,  and  graduate  students  from 
three  different  universities  (Northwestern,  UCLA,  and  Stanford);  clerical 
workers  from  two  Southern  California  companies;  student  nurses  from  a 
Chicago  hospital;  two  classes  of  elementary  school  children  (third  and 
sixth  grades);  Mexican-American  residents  of  "East  L.A.";  and  two  groups 
of  Cuban  exiles  who  now  reside  in  Southern  California.  In  many  cases, 
multiple  samples  were  drawn  from  larger  populations  and  a  different  experi¬ 
mental  treatment  was  applied  to  each  sample  group.  A  summary  of  the 
populations  used  in  research  completed  under  this  contract  is  contained 
in  Table  1.  This  table,  however,  does  not  reflect  the  actual  number  of 
samples  tested,  experimental  treatments  applied,  or  subcultures  studied 
(a  single  population  may  have  several  subcultures  operating  simultaneously). 
For  convenience  of  presentation  this  detailed  information  has  been 
included  in  Appendix  I. 


TABLE  1 

SUMMARY  OF  SUBJECT  POPULATIONS  AND  EXPERIMENTAL  TREATMENTS  USED  10  FROJECT 

ECHO,  PHASE  I 


Population 

Code 

Number  of 

Ss  in 

Population 

Number  of 

Classifications 

Completed 

Message 

Session 

Held? 

No. 

Ss 

Number 

of 

Forms 

UCLA 

B 

137 

17 

Yes 

70 

(2) 

Northwestern 

T 

68 

11 

Yes 

83 

(6) 

Stanford 

A 

72 

2 

No 

Nurses 

N 

52 

12 

Yes 

37 

(2) 

Carnation 

Company 

C 

54 

2 

Yes 

31 

(2) 

Prudential 

Company 

P 

93 

3 

Yes 

98 

(2) 

Third  Graders 

3 

17 

3 

No 

Sixth  Graders 

6 

24 

3 

No 

Cuban  Exiles 

CE/K 

25/18 

10/8 

Yes 

25 

(2) 

Total 

560 

71 

387 

16 
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V.  DATA  COLLECTION 

The  ECHO  question  can  be  cast  in  a  variety  of  forms  according  to 
the  needs  of  the  investigators;  different  forms  tap  different  attitudes. 
There  are  five  major  variables  in  the  question: 

1.  Role  assignment  (p) :  the  socially  prescribed  position  to  be 
used  as  the  frame  of  reference  for  answering  the  question 

2.  Event  (e) :  the  kind  of  occurrence,  usually  a  behavior, 
solicited 

3.  Event  evaluation  (n) :  the  positive  or  negative  quality  of 
the  event  (behavior) 

4.  Reinforcement  (tt)  :  specification  of  either  positive  or 
negative  reinforcement 

5.  Source  (a):  the  person(s)  or  concept(s)  which  provides  the 
reinforcement . 

In  some  cases  a  sixth  variable  may  be  introduced  to  define  the 
relationship  between  the  role  holder  (p)  and  another  role,  role  holder, 
or  group. 


For  example,  a  question  designed  to  tap  the  areas  of  conflict 
between  segments  of  a  society  might  take  the  following  form: 


What  is  something  a  person  like  you  (p)  could  do  (e)  that 
£p,ur  friends  would  say ,  was_  foolish  (n)  but  that  someone  else 

would  graise  (tt)  you  for  doing?  Who  would  that  someone  else 
\o)  be?  - 


Two  forms  of  the  projective  question  have  been  used  to  collect  data 
during  the  Phase  I  research:  "What  is  a  (good/bad)  thing  to  do?"  and 
"What  is  a  (pood/bad)  thing  that  could  happen?"  The  latter  question  was 
used  during  the  summer  1966  pilot  studies  and  has  been  given  only  cursory 
consideration  during  this  phase  of  the  research. 
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Subjects  were  asked  Co  write  their  answers  on  IBM  cards,  which 
became  the  basic  data  for  all  subsequent  operations. 
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VI.  CLASSIFICATION 

The  classification  process  is  an  integral  part  of  the  ECHO  methodology. 
It  is  based  on  the  hypothesis  that  indigenous  classifiers  (members  of  the 
exact  population  under  study)  are  able  to  make  fine  discriminations  among 
statements  from  their  own  subculture  and  to  understand  the  nuances  of  the 
language,  and  that  they  do  this  better  than  "experts".  This  hypothesis 
was  supported  in  the  multiple  classification  studies,  reported  below,  and 
in  the  validity  test  sessions  (Sec.  VII). 

A.  METHOD 

The  classifiers  worked  first  individually,  each  with  one-third  of 
the  deck,  sorting  the  cards  into  categories  that  had  meaning  to  them.  The 
investigator  did  not  give  examples  of  categories  since  any  example  would 
tend  to  structure  the  process  in  terms  of  tb'  investigator’s  frame  of 
reference;  ECHO  is  interested  in  discovering  the  frames  of  reference  of 
the  subject  population. 

Although  classifiers  take  varying  amounts  of  time  for  this  step, 
the  modal  time  is  approximately  30  min.  After  all  three  had  completed 
the  individual  sorting,  they  joined  together  to  develop  a  single  set  of 
categories  to  include  all  cards.  One  person  read  the  cards  in  one  of  his 
stacks  and  the  others  added  cards  that  meant  the  same  thing.  The  process 
was  continued,  with  the  other  two  taking  turns  reading  cards,  until  all 
the  cards  had  been  placed  into  categories.  They  then  titled  the  categories, 
either  making  up  titles  or  using  a  representative  answer  found  on  one  of 
the  cards  in  that  category . 

The  investigator  assigned  a  two-digit  code  number  to  each  category, 
the  cards  were  punched,  and  then  they  were  classified  by  another  team. 

B.  QUANTITY  OF  DATA 

A  team  can  comfortably  classify  the  data  c?rds  of  25-30  subjects, 
that  is,  250-300  cards. 
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This  conclusion  was  reached  by  a  process  of  successive  approximations. 
The  first  classification  session  at  UCLA  was  designed  to  test  the  process 
as  well  as  to  generate  data.  Two  teams  each  classified  approximately  1400 
cards.  Two  sessions,  six  hours  and  five  hours  long,  were  required.  The 
high  motivation  and  enthusiasm  that  seemed  clear  at  the  beginning  changed 
to  discomfort,  fatigue,  boredom,  and  frustration  as  the  hours  passed. 

In  the  studies  at  two  Southern  California  companies,,  the  classifica¬ 
tion  teams  worked  with  approximately  500  cards  (516,  526,  599,  600  and 
472  cards) .  Although  fatigue  and  boredom  were  not  as  pronounced  as  in 
the  UCLA  sessions,  the  investigators  were  still  impressed  by  the  decreased 
efficiency  as  time  passed. 

In  the  Stanford  study,  the  two  classification  teams  each  worked  with 
700  cards.  One  team  completed  its  task  in  approximately  5  hours;  the 
other  team  stopped  work  at  the  end  of  4-1/2  hours,  and  completed  the  task 
in  3  hours  the  following  morning. 

In  the  Northwestern  study  the  classifiers  worked  with  decks  of 
approximately  325  cards.  The  Nurses  study  classifiers  worked  with 
approximately  250  cards.  In  both  groups,  a  complete  classification 
took  from  2  to  3  hours.  The  team  members  did  not  appear  fatigued;  informal 
interview  data  supported  the  observation.  In  both  studies,  a  second  set 
of  four  teams  reclassified  the  data  two  days  later  under  virtually  identical 
conditions.  Although  the  second  group  had  had  no  experience  with  the 
process,  the  time  to  complete  the  task  varied  from  1-1/2  to  2-1/2  hours. 

A  possible  explanation  for  this  time  reduction  is  that  the  second  group 
learned  from  classmates  that  the  task  was  nonthreatening  and  therefore 
approached  it  with  more  confidence. 

In  the  course  of  the  UCLA  multiple  classification  session,  described 
below,  11  classification  sessions  with  150  cards  were  run.  The  modal 
time  was  under  two  hours;  only  one  group  took  3  hours.  Four  sessions 
using  300  cards  were  completed  in  2-1/2  to  3  hours  . 
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The  Cubar  study  classifiers  worked  with  packets  of  90,  100,  110, 
and  140  cards.  During  each  three-hour  session  each  group  classified 
twice . 


C.  MULTIPLE  CLASSIFICATION  OF  DATA 

Current  findings  indicate  that  a  given  deck  of  cards  should  be 
classified  by  three  different  teams  to  give  a  measure  of  reliability. 
Also,  the  categories  from  one  classification  can  be  used  to  interpret 
the  categories  of  another. 


Multiple  classification  appears  to  be  one  of  the  most  powerful 
tools  for  data  analysis.  The  process  is  simple:  two  or  more  teams 
independently  classify  the  same  set  of  cards.  Studies  conducted  with 
UCLA,  Northwestern,  Nurses  and  Cuban  groups  indicated  that  when  classifiers 
came  from  the  same  groups,  reliability  tended  to  be  high  (i.e.,  the  same 
cards  were  grouped  together),  but  these  same  teams  used  different  numbers 
of  categories  and  different  category  widths.  Examination  of  the  structures 
of  the  categories  indicated  that  when  the  cards  placed  in  a  large  category 
by  one  group  were  distributed  into  two  or  more  categories  by  another 
group,  the  category  labels  had  semantic  equivalence. 

Example:  The  category  "Help  other  people"  from  UCLA  female 
(FI)  classifiers  was  distributed  by  another  group  (F2)  of 
classifiers  into  three  categories  with  titles  "Help  someone 
at  your  own  expense,"  "Help  those  less  fortunate,"  and  "Help 
others."  Since  both  the  language  and  culture  were  familiar, 
it  was  possible  to  state  with  some  assurance  that  the  classi¬ 
fiers  were  in  agreement,  but  that  the  second  group  made 
finer  discriminations. 


When  classifiers  are  drawn  from  somewhat  different  populations  the 
classifications  have  less  overlap,  presumably  the  result  of  a  difference 
in  social  perceptions.  Table  2  shows  male  data  classified  by  a  team  of 
male  (M2)  and  a  team  of  female  (F4)  classifiers.  An  examination  of  the 
titles  indicates  that  they  are  not  in  disagreement  but  only  organized 
differently.  Thus,  the  male  concept  of  "Be  a  better  person  and  relate  to 
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TABLE  2 

UCLA  MALE  (M2)  DATA  CLASSIFIED  BY  F4  and  M2 

Ig55le.jF,4)  Classifications  Male  (M2)  Classification 

"Live  up  to  self-ideal"  (f  =  7) 

"Be  more  easy-going"  (f  .  4)  "Be  a  better  person  and  relate 

"Have  more  self-confidence"  (f  .  3)  to  my  environment  better”  (f  -  21) 

"Realize  minor  goals"  (f  =  3) 

"Be  more  liveable"  [sic]  (f  =  3) 

"Improve  sphere  of  knowledge"  (f  =  1) 

my  environment  setter"  meant,  to  these  women,  a  number  of  things:  "Live 
up  to  self-ideal",  "Be  more  easy-going",  "Have  more  self-confidence", 

"Realize  minor  goals",  "Be  more  liveable"  [sic],  and  "Improve  sphere  of 
knowledge".  A  person  from  our  culture  can  intuitively  see  the  logic  of 
that  particular  breakdown  of  the  larger  category.  Two  groups  such  as 
this,  males  and  females  from  the  same  subculture,  understand  each  other 
quite  well:  a  number  of  different  groups  have  checked  the  classifications 
made  by  other  groups  to  see  if  they  would  agree  with  the  way  the  cards 
were  sorted;  other  groups  were  given  the  category  titles  and  asked  to  sort 
the  cards  into  those  categories.  The  agreement  was  almost  absolute.  The 
multiple  classification  method  appears  potentially  fruitful,  and  the 
method  itself  is  undergoing  refinement. 

D.  ARE  INDIGENOUS  CLASSIFIERS  NECESSARY? 

All  evidence  indicates  that  indigenous  classifiers  (Cs)  are  more 
able  than  experts  to  organize  the  raw  inputs  into  meaningul  categories; 
this  ability  accrues  from  the  indigenous  classifiers'  greater  familiarity 
with  the  subject  population. 
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Example:  One  group  of  University  Cs  distributed  a  batch  of 
cards,  which  the  experts  (a  professor  and  a  graduate  student) 
defined  as  "Be  kind  to  roommates,"  into  two  different  cate¬ 
gories:  "Be  kind  to  dorm  roommates"  and  "Be  kind  to  apartment 
roommates."  In  explaining  their  actions  the  £s  pointed  out 
that  apartment  roommates  arc  secured  through  a  "self-selection" 
process  while  the  dormitory  roommates  are  assigned  by  an 
impersonal  computer;  therefore,  more  tolerance  and  understanding 
is  due  the  former. 

The  importance  of  indigenous  classifiers  is  evident  when  the  work 
of  alien  Cs  is  compared  with  the  classifications  made  by  indigenous  £s. 

For  example,  a  batch  of  115  UCLA  cards  was  classified,  at  different  tiroes, 
by  a  group  of  three  UCLA  coeds  and  a  group  of  three  female  Cubans  .  The 
categories  created  by  each  team  and  the  cards  composing  each  of  these 
categories  are  shown  in  matrix  form  in  Table  3.  Obviously,  the  UCLA  £s 
were  able  to  make  finer  discriminations;  the  shaded  area,  for  instance, 
shows  that  the  Cubans  lumped  together  five  UCLA  categories  ("study  hard 
to  get  good  grades";  "prepare  for  future  career";  "graduate  in  alloted 
time;"  "learn  for  learning's  sake;"  and  "learn  new  skills")  into  a  single 
general  category,  "learning  and  get  good  grades."  The  matrix  has  other 
examples  of  this  discrimination  differential.  Experimentation  with  both 

I 

Cuban  and  College  £s  indicates  that  these  apparent  differences  in  social 
perceptions  are  not  artifacts  of  culture-specific  ways  of  categorization 
(e.g.,  using  few  or  many  categories),  but  represent  different  ways  of 
viewing  the  same  concepts,  which  is  what  one  might  expect  between 
cultures. 


E.  CONCLUSIONS 

1.  The  optimum  number  of  cards  for  one  classification  session  is 
250-300.  (Note  that  investigations,  reported  below,  indicate 
that  this  number  yields  a  stable  data  base.) 

2.  If  larger  numbers  are  to  be  classified,  the  sessions  should 
be  broken  so  that  no  session  is  longer  than  three  hours.  A 
minimum  of  two  hours  should  elapse  between  sessions. 

3.  Each  deck  should  be  classified  three  times. 
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TABLE  3 

EXAMPLE  OF  TWO  CLASSIFICATIONS  OF  SAME  DATA 

"Eyeball  Analysis"  Indicates  That  the  UCLA  Cs  Were  More  Able  to  Make  Finer 
Distinctions  Among  Concepts.  Shaded  Areas  Illustrate  Differences  in  Two 

'Teams'  Social  Perceptions. 

.  JS 
«  0. 

T3  O 

a  •» 


UCLA  £ 

FEMALES  CLASSIFYING  F2 

Prepare  for  Future  Career 
Do  Chorea 

Help  a  Friend  With  a  Problem 
Be  Sociable 

Be  Nice  to  Family  Membera 
Be  Tolerant  with  Family  Membera 
Be  a  Good  Parent 

Be  Kind  and  Considerate  of  Othera 
Experience  Life  Fully 
Improve  oneaelf  (Qualities) 

Study  Hard  to  Git  Good  Gradea  >' 
Graduate  in  Allotted  Time 
Learn  for  Learning' a  Sake 
Learn  New  Skllla 
Get  Married 

Participate  in  Organliationa 
(extracurricular  actlvltlea) 

Miscellaneous 
Co  to  Church 
Be  Attractive 
Attend  Cultural  Eventa 

Break  lad  Habita 


9  10 

1 

4 

10 

H 

© 

H  ... 

11 

V 

4 

2  12 

6 

20  12 


20  17  4 
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Each  classification  should  be  checked  by  having  another  team 
sort  the  cards  into  the  categories  generated  by  the  original 
team. 

The  classifiers  (including  the  team  mentioned  in  (4)  above) 
should  rank  the  categories  for  importance  (or  whatever  variable 
is  being  investigated).  Note  that  this  is  a  check  on  ranking 
by  frequency. 
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VII.  VALIDITY  CHECK  ~  ^ 

As  an  estimate  of  validity  ("Does  the  instrument  do  what  it  purports 
to  do?"),  test  sessions  were  run  to  see  if  ECHO-generated  data  represented 
important  and  prevalent  values  in  the  subject  culture.  The  sessions  also 
allowed  the  assessment  of  the  method's  sensitivity:  could  it  distinguish 
among  similar  subcultures?  In  addition,  the  test  sessions  were  used  to 
determine  how  to  select  specific  value  statements  from  the  many  categories 
supplied  by  the  classifiers.  Six  groups  were  tested:  UCLA,  Carnation, 
Prudential,  Northwestern,  Nurses,  and  Cuban  Exiles. 

A.  PROCEDURE 

Although  each  session  differed  in  some  way  from  the  others,  the 
same  basic  procedure,  a  "message  session",  was  followed  in  all.  The 
subjects  were  given  test  booklets  that  contained,  in  paired-comparison 
format,  equal-length  "messages"*  (lists  of  category  titles)  that  came 
from  three  sources:  (1)  classification  sessions  that  represented  the 
subject  population;  (2)  classification  sessions  of  other  populations;  and 
(3)  a  list,  in  the  same  form,  of  items  prepared  by  an  "expert"  in  that 
culture.  Ss  were  required  to  select  the  one  of  each  pair  that  they 
believed  to  be  the  "most  important".  Figure  2  is  a  sample  page  from  a 
message-session  test  booklet. 

The  position  of  each  pair  of  messages  on  a  page  was  alternated  and 
the  pages  were  placed  in  random  order  in  the  test  booklets  to  control 
position  and  ordering  effects.  Each  message  from  a  classification  session 

consisted  of  the  titles  of  the  five  categories  with  the  highest  frequency, 
in  rank  order. 


In  some  cases,  special  test  booklets  were  prepared  for  different 
subgroups.  For  example,  the  Carnation  subjects  were  divided  into  two 
groups,  secretaries  and  non-secretaries,  and  different  messages,  based 


refer 6to  thTlSf  ^  an  early  Study5  ±fc  is  now  used  to 

the  list  of  words  or  phrases  used  in  a  validity  check  session. 
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Figure 
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■  R60 

HAVE  FUN  AND  GET  ENJOYMENT  OUT  OF  LIFE 
MAKE  OTHERS  HAPPY 

SET  PERSONAL  MORALS  AND  OBEY  THEM:  BE  INTROSPECTIVE 
AND  SELF-CRITICAL 

WORK  HARD  AND  TO  ONE'S  FULLEST 

HELP  SOMEONE  IN  TROUBLE 


Ml 

TO  EXPRESS  ONE'S  TRUE  OR  IDEAL  FEELINGS 

ACHIEVE  FUTURE  GOALS 

BE  CONCERNED  WITH  EDUCATION 

HELP  LESS  FORTUNATE  PEOPLE 

LIVE  AND  ENJOY  LIFE 


WRITE  YOUR  ANSWER  HERE _ .  Go  on  to  the  next  pa^e 


.  Sample  Page  From  a  Northwestern  Test  Session  Booklet.  R60 
is  From  Northwestern  Female  Data;  Ml  is  From  UCLA  Male  Data 
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on  secretarial  and  non-secretarlal  data  respectively,  were  prepared  for 
them.  Similarly,  the  male  subjects  in  the  Cuban  and  Northwestern  groups 
received  test  booklets  containing  messages  based  mainly  on  the  male 
responses  while  the  females  in  those  two  groups  received  messages  created 
mainly  from  female  responses. 

The  hypothesis,  as  stated  earlier,  was  that  Ss  would  select  the 
list  which  came  from  their  own  population  over  any  other,  and  would  select 
lists  which  came  from  a  similar  population  over  lists  coming  from  "alien" 
populations.  "Population",  in  this  instance,  refers  to  both  the  data 
source  and  the  classifiers.  Table  4,  in  the  example  below,  shows  some 
of  the  possible  variations  of  classifier  and  data-source  mixes. 

B.  MESSAGE  CONSTRUCTION 

Several  decisions,  based  on  the  judgment  of  the  investigators,  were 
made  about  message  construction:  (1)  the  language  of  the  subjects  should 
be  used  unchanged;  (2)  the  selection  of  the  specific  items  to  be  included 
should  be  by  mechanical  means  and  independent  of  the  judgment  of  the 
investigators;  (3)  messages  should  contain  approximately  five  items.  In 
accordance  with  the  first  decision,  no  connecting  words  or  phases  could 
be  used,  so  category  titles  were  presented  in  a  list.  To  implement  the 
second  decision  several  methods  were  tried;  at  the  present  time  the 
evidence  seems  clear  that  the  categories  with  the  highest  frequencies 
(the  largest  number  of  cards)  are  the  most  representative.  Appendix  II 
contains  data  supporting  that  contention.  The  other  methods  tried  were 
(1)  ranking  by  classifiers  (high  correlation  with  frequency) ,  (2)  ranking 
by  S's  estimate  of  importance,  and  (3)  selection  by  source;  it  is  possible 
that  this  last  nethod  may  have  merit  for  other  research  purposes. 

The  t\iir\  decision,  to  limit  the  number  of  categories,  fit  the 
empirical  riud'.ngs:  in  almost  every  classification  session,  the  five 
categories  wxth  the  highest  frequencies  accounted  for  over  60%  of  the 
cards.  In  the  few  exceptions,  the  sixth  category  was  tied  with  the  fifth. 
When  a  tie  occurred,  a  coin  was  tossed  to  determine  the  rank. 
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An  expert  was  asked  to  write,  in  rank  order  of  importance,  the 
significant  components  of  the  clerical  jobs  in  his  company.  An  expert 
was  defined  as  being  a  person  with  considerable  knowledge  about  a 
population  but  wl.o  was  not  part  of  that  population.  An  example  was  the 
personnel  manager  of  Prudential  relative  to  the  clerical  force  of  that 
company. 

C.  EXAMPLE  OF  RESULTS 

The  details  of  each  validity  check  session  will  be  found  in  Appendix 
I;  the  Cuban  study  is  reported  here  as  an  example. 

1.  List  Construction 

The  Cuban  lists  consisted  of  the  titles  of  the  five  highest-frequency 
categories  in  each  of  six  classification  sessions.  Table  4  shows  the 
classification  systems  used. 


TABLE  4 

CUBAN  CLASSIFICATION  SYSTEMS  EMPLOYED  IN  CONSTRUCTION  OF  LISTS  FOR  VALIDITY 

TEST 


LIST 

I 

III 

V 

VI 

VII 
IX 


CLASSIFICATION  SYSTEM 

Males  classifying  male  positive  data 

Females  classifying  female  positive  data 

Males  classifying  combined  male  positive  and  negative  data 

Females  classifying  combined  female  positive  and  negative 

data 

Males  classifying  female  positive  data 
Females  classifying  male  positive  data 


In  addition,  a  list  (translated  into  Spanish)  was  constructed  from 
the  five  hignest-frequency  categories  in  the  Stanford  positive  data. 
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2.  Procedure 

The  lists  were  presented  In  a  paired-comparison  format,  the  Ss 
being  instructed  to  select  from  each  pair  the  list  which  they  considered 
to  be  "most  important". 

The  test  booklets  consisted  of  randomly  ordered  pages,  each  of 
which  contained  two  lists  and  an  answer  blank.  Figure  3  is  a  sample  page. 
Male  and  female  j>s  received  two  slightly  different  test  booklets. 

3.  Results 

Table  5  indicates  the  proportions  of  Ss  selecting  each  of  the  lists 
and  the  levels  of  significance  of  the  results. 


XI 

AYUDAR  A  OTROS. 

INCREMENTAR  EL  BIENESTAR  COMUN. 
ESTAR  INTERESADO  POR  MUCHAS  COSAS. 
VIVIR  LA  VIDA  PLENITUD. 

ACTIVIDAD  FAMILIAR. 


Ill 

AYUDAR  AL  PROJIMO. 

LIBERACION  DE  CUBA. 

PROPAGACION  DE  LA  FE  CRISTIANA. 

RESPETO  A  LAS  LEYES  Y  BUEN  COMPORTAMIENTO  EN  ESTE  PAIS. 
COMBATIR  EL  COMUNISMO. 


RESPUESTA 


Figure  3.  Sample  Page  From  Cuban  Test  Booklet.  XI  is  a  Translation  of 

Stanford  Data;  III  is  Cuban  Female  Data 
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Ss  appeared  to  discriminate  between  the  lists  on  the  basis  of  the 
sex  and  the  culture  of  the  subjects  from  whose  responses  the  lists  were 
generated.  For  example,  though  the  male  Ss  rejected  their  own  list  (I) 
in  favor  of  that  generated  by  the  Cuban  females  (III),  they  significantly 
chose  their  own  lists  over  that  based  on  the  Stanford  data  (XI).  These 
results  also  present  some  evidence  on  the  question  of  whether  positive 
and  negative  responses  should  be  classified  together  or  separately.  As 
can  be  seen  in  Table  5,  both  male  and  female  subjects  selected  their  own 
positive  lists  over  those  generated  by  the  classification  of  combined 
positive  and  negative  responses . 

The  validity  test  results  indicate  tht  the  ECHO  instrument  can 
determine  not  only  important  attitudes  within  the  subject  population  but 
also  m^ny  of  the  differences  among  various  subgroups  within  the  population. 
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VIII>  POWER  STRUCTURE  ANAT.VS  t g 

The  question,  "What. is  a  good/bad  thing  to  do?"  has  a  second 

part:  "Who  would  approve?"  The  answers  tc  this  question,  which  we  call 
"sources"  and  sociologists  call  'surrogates,"  provide  additional  infor¬ 
mation  about  the  perceived  "reinforcers"  which  control  behavior  in  the 
subject  culture.  A  tenable  assumption  is  that  human  beings,  like  all 
animal  life  from  one-celled  amoebae  to  Nobel-prize-winning  physicists, 
tend  to  seek  positive  reinforcement  (reward)  and  avoid  negative  reinforce¬ 
ment  (punishment);  those  who  dispense  these  reinforcements  hold  "power" 
in  the  culture.  People  acquire  the  beliefs,  attitudes,  norms,  and  values 
of  their  reference  culture  through  a  process  of  learning,  i.e.,  by 
experiencing  the  positive  and  negative  consequences  of  their  behaviors. 
This  learning  process  is  referred  to  as  "socialization." 


The  source (s)  which  control  a  behavior  may  change  over  time.  In 
most  cases,  the  source  of  approval  or  disapproval  changes  from  an  external 
entity  o  an  internal  control  mechanism,  i.e.,  people  learn  to  feel  guilty* 
when  they  do  something  that  their  society  says  is  wrong.  The  Power  Struc¬ 
ture  Analysis  element  of  the  ECHO  methodology  permits  identification  of  the 
perceived  sources  of  reinforcement  and  changes  in  the  struct  ,  or  pattern, 
of  power  impinging  on  the  individual.  Our  studies  indicate,  for  instance,  ’ 
that  children  in  the  third  grade  tend  to  view  their  parents,  particularly 
mother,  as  the  most  important  source  of  approval  and  disapproval.  College 
students,  on  the  hand,  cite  self'  as  the  most  important  source.  The 
college  students  appear  to  have  "internalized"  the  value  system  of  the 
culture;  the  third  graders  still  use  their  parents  to  tell  them  what 
society  thinks  is  right  or  wrong.  Sixth  grade  children  respond  more  like 
the  college  students.  Figure  4  shows  the  relative  degree  of  influence 


dnne  that  "fullt"  is  w|]at  a  Person  experiences  when  he  knows  that  he  has 

iout8?"'  ill  ZTV  u™  lB  ”hat  he  feels  lf  “J-one  •!»  knows 
about  it.  The  two  feelings  are  not  necessarily  the  same. 
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Figure  4.  Relative  Influence  of  "Self"  and  "Parent*"  At  Three  Grade  Levels 
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that  "self"  and  "parents"  exercise  over  the  behaviors  of  the  individuals 

in  our  test  populations;  the  increasing  frequency  of  the  term  "self"  may 

well  reflect  the  process  of  "internalization"  (or  in  sociological  terms 

* 

"interiorization") . 

Implications  for  behavior  modification  immediately  suggest  themselves. 
For  instance,  data  collected  so  far  indicates  that  the  parents,  and  partic¬ 
ularly  the  mother,  are  overwhelmingly  the  significant  figures  in  the  life 
of  a  child.  It  follows  that  if  one  wishes  to  be  effective  in  modifying 
the  behavior  of  a  disturbed  child,  he  must  deal  with  the  parents,  either 
by  minimizing  their  impact  or  modifying  their  behavior.  This,  indeed,  is 
what  child  therapists  do:  mere  time  is  spent  with  the  parents  of  a 
disturbed  child  than  with  the  child  himself.  Or,  looking  at  the  Cuban 
data,  one  might  hypothesize  that  if  the  behaviors  of  Cuban  women  are  to 
be  modified,  the  Church  would  be  a  potent  force;  whereas  for  Cuban  men  it 
would  be  less  so.  Evidence  for  this  statement  comes  from  the  frequent 
mention  by  Cuban  females  of  church  and  church-related  reinforcers. 

The  Power  Structure  Analysis  is  also  capable  of  detecting  cultural 
differences  in  sources  of  reinforcement.  For  instance,  "Anglo  university 
students,  both  male  and  female,  cite  "myself"  as  source  of  approval  far 
more  often  than  Cuban  subjects  do  (see  Table  6).  Note  too  that  males,  in 
both  cultures,  cite  "myself"  more  often  than  do  their  female  counterparts; 
the  degree  of  difference  between  males  and  females  on  this  dimension  might 
be  thought  of  as  an  index  of  the  cultures'  belief  in  the  "equality"  of  the 
masculine  and  feminine  roles. 


*  The  comments  on  "value  internalization"  and  the  "socialization  process 
are  based  on  the  findings  of  our  research  with  two  populations  of  ele¬ 
mentary  school  children.  These  experiments  were  designed  to  answer  two 
major  questions:  (1)  How  old  must  our  subjects  be  to  use  the  standard 
IBM  card  form  of  the  instrument?  and  (2)  Can  we  detect  the  changes  which 
accompany  (or  constitute)  the  socialization  process?  A  complete  report 
of  this  phase  of  the  research  is  contained  in  Appendix  III. 

**The  literature  is  replete  with  evidence  that  supports  this  hypothesis; 
e.g.,  see  Area  Handbook  for  Cuba,  Special  Operations  Research  Office. 
American  University,  1961,  p.  138. 
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TABLE  6 

FREQUENCY  OF  OCCURRENCE  OF  "MYSELF"  AS  SOURCE  OF  APPROVAL  OR  DISAPPROVAL 


Ref. 


Code 

Subject  Population 

Total  N 

f  of  "Myself 

%  of  N  "Myself 

A1 

Stanford  Positives 

700 

258 

36.9 

A2 

Stanford  Negatives 

695 

226 

32.5 

B+ 

UCLA  Positives  (all  Ss) 

1169 

387 

33.1 

B- 

UCLA  Negatives  (all  Ss) 

NOT  AVAILABLE 

B+ 

UCLA  Female  Positives 

506 

151 

29.8 

B+ 

UCLA  Male  Positives 

663 

235 

35.4 

CE1 

-  CUBAN  DATA 

I  &  III 

Cuban  Positives  (all  Ss) 

259 

36 

13.9 

II  &  IV 

Cuban  Negatives  (all  Ss) 

241 

41 

17.0 

I 

Cuban  Male  Positives 

150 

31 

20.7 

II 

Cuban  Male  Negatives 

140 

24 

17.1 

III 

Cuban  Female  Positives 

109 

5 

4.6 

IV 

Cuban  Female  Negatives 

101 

17 

16.8 

A  behavior  may  be  reinforced  by  two  completely  different  sources 
in  the  same  culture,  and  the  ECHO  power  structure  analysis  can  detect 
this  fact.  For  example,  the  UCLA  students  saw  "getting  married"  as  a 
good  thing  to  do,,  When  asked,  "Who  would  approve?",  the  men  said  "myself" 
and  the  women  said  "my  parents"  (see  Table  7) . 


TABLE  7 

UCLA  STUDENTS:  "IT'S  A  GOOD  THING  TO  GET  MARRIED" 

Gender  of  Ss 


SOURCE  OF  APPROVAL 

MALES 

FEMALES 

FAMILY 

1 

10 

11 

MYSELF 

9 

1 

10 

10 

11 

21 

13.61  with  v  ■  1  is  significant  at  a  ■  .001+ 
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The  Power  Structure  Analysis  element  of  ECHO  technique  appears  to 
be  a  powerful  toox  for  describing  and  understanding  populations.  The 
usefulness  of  this  review  will  be  greatly  increased  in  the  future  when 
the  relationship  between  specific  values  and  sources  is  better  understood. 
Subsumed  in  this  relationship  is  an  understanding  of  which  values  go 
together  in  natural  clusters  or  constellations.  Exerting  pressure  for 
change  on  one  element  of  a  cluster  will  cause  a  reaction  in  the  other 
elements  of  the  cluster;  understanding  the  relationship  between  values 
will  permit  greater  accuracy  in  predicting  change  (and  therefore  greater 
accuracy  in  influencing  change).  Sourcetvalue  relationships  are 
endogenous  to  this  value  cluster  problem. 
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IX.  DATA  PROCESSING 

This  section  is  divided  into  three  subsections,  each  of  which  treats 
a  different  facet  of  the  data  processing  procedures  used  in  Phase  I.  The 
first  subsection  explicates  the  "Operational  Package"  which  is  used  to 
transform  the  raw  inputs  into  value  hierarchies  and  "power  structures"; 
the  second  subsection  lists  the  library  programs  which  have  been  developed 
for  general  use  under  the  terms  of  the  ECHO  contract.  Possible  applications 
of  recent  advances  in  computer  technology  are  considered  in  the  last 
subsection. 

Automatic  data  processing  by  high-speed  computers  is  an  important 
element  of  the  ECHO  methodology  which  allows  reliable,  unbiased  data 
analysis  while  minimizing  time  and  cost.  All  of  the  computer  operations 
in  the  project  were  completed  on  a  CDC  3600  computer,  operated  by  the 
Computer  Center  of  Santa  Barbara,  a  General  Research  Corporation  subsidiary. 
The  computer  programs  used  in  Phase  I  of  Project  ECHO  are  of  two  types: 
Operational  Package  Programs  and  Libraiy  Programs. 

A.  "OPERATIONAL  PACKAGE"  PROGRAMS 

Each  of  these  39  programs  performs  the  standard  analytical  computa¬ 
tions  that  all  raw  ECHO  inputs  undergo  as  part  of  the  "Operational  Package" 
processing. 

KOUNTEM — computes  the  frequency  of  all  category: source  combinations. 
The  program  was  modified  to  allow  comparison  of  different  classifications 
of  the  same  data  cards.  Each  of  the  eight  versions  provides  information 
about 

1.  The  number  of  cards  (N)  in  the  data  pool 

2.  The  number  of  responses  in  each  category,  i.e.,  the  prevalence 
of  the  value  in  the  hierarchy  of  the  target  populations 

3.  The  relative  importance  of  each  source  associated  with  a  given 
value 
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TRANSVES — compares  any  two  classification  systems  which  are  based, 
in  whole  or  in  part,  on  the  same  set  of  responses.  The  output  from  this 
program  answers  the  question,  "What  did  two  different  groups  of  Ss  say 
about  the  same  cards?"  This  program  has  28  versions  now  in  operation. 

SUBJS0UR-— identifies  the  frequency  of  sources  cited  by  each  subject. 
Information  from  this  output  can  be  used  in  answering  questions  about  the 
power  structure  impinging  on  any  g  ,.ven  subject  in  the  population. 

POWERSTRC — quantifies  the  relative  "power"  each  source  has  over  a 
given  behavior  in  the  population's  "collective  repertoire." 

Example:  Male  and  female  college  students  expect  different  people 
to  approve  if  they  "get  married,"  as  the  following  POWERSTRC  output 
demonstrates.  (Table  7,  above,  is  an  abridgement  of  this  output.) 


Source  of 
Approval 

oa  vji 

Male 

CilUCL 

Female 

Myself 

9 

1 

Parents 

1 

10 

Mate 

2 

- 

Society 

1 

- 

Totals 

13 

11 

WRECKEM — Computes  the  frequency  of  every  combination  of  descriptions 
assigned  daring  several  classifications  of  the  same  data.  This  process 
facilitates  the  comparison  of  several  groups'  opinions  of  the  same  cards 
in  cases  where  more  than  two  teams  classified  the  same  data.  Representing 
the  data  in  matrix  form  when  there  are  more  than  two  nominally  scaled 
dimensions  is  not  feasible. 
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B.  LIBRARY  PROGRAMS 

Several  other  programs  were  written  especially  for  the  CDC  3600 
computer  used  in  the  data  analysis  portion  of  the  project.  This  library 
includes  the  following  programs : 

1.  Central  Tendency  Statistics 

a.  mean  (x) 

b .  mode 

c.  median 

2.  Dispersion  (Variability)  Statistics 

a.  sample  variance 

b.  sample  standard  deviation 

c.  third  moment  about  the  mean 

d.  single  factor  analysis  of  variance 

3.  Correlation  Statistics 

a.  Pearson  Coefficient  of  Correlation  (p) 

b.  Spearmen's  Rho  (p)  for  Ranked  Data 

c.  Kendall  W  for  Ordinal  Data 

A.  Chi-Square  Tests 

a.  r*K  multifactor  Chi-Square  Test  with  (r  -  1) (K  -  1) 
degrees  of  freedom 

b.  Single-sample  Chi-Square  Test  with  (K  -  1)  degrees  of 
freedom 

5 .  Student's  't'  Test"  for  Sample  and  Population  Data 

C.  OTHER  POSSIBLE  APPLICATIONS  OF  COMPUTER  TECHNOLOGY  TO  THE  ECHO 
METHODOLOGY 

Three  computer  innovations  have  been  considered  as  being  of  possible 
value  to  the  ECHO  methodology:  (1)  Computerizing  the  classification  process; 
(2)  generating  the  exact  vocabulary  of  our  subjects  in  concordance  form; 
and  (3)  identifying  the  common  syntactical  style(s)  of  the  population. 

The  latter  two  ideas  hold  promise  as  possible  beginnings  of  a  computerized 
persuasive  message  generation  system. 
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The  possibility  of  computerizing  the  classification  system  has 
been  investigated  and  rejected.  .  Indigenous  classifiers  appear  to  be  an 
important  element  of  the  ECHO  methodology;  even  without  this  technical 
constraint,  the  cost  of  such  an  operation  would  be  prohibitive. 
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X.  SUMMARY  OF  MAJOR  PHASE  I  FINDINGS 

A  brief  synopsis  of  some  Phase  I  results  follows. 

A.  SAMPLE  SIZE  REQUIREMENTS 

Outputs  from  15-20  people  have  been  used  to  accurately  reflect  the 
value  structure  of  a  100  to  200  person  group.  Insofar  as  this  larger 
group  is  representative  of  a  still  larger  group  (a  university,  ten-year 
old  children,  a  country),  the  very  small  sample  can  yield  information,  in 
a  brief  period  of  time,  formerly  available  only  from  experts  who  had 
spent  years  studying  that  population.  ECHO  information  is  current;  the 
"expert"  may  be  obsolete  without  knowing  it.  The  accuracy  of  the  general¬ 
ized  results  varies  as  a  function  of  target  group  homogeneity,  represent¬ 
ativeness  of  the  sample,  motivation  and  fatigue  of  the  indigenous  classi¬ 
fiers,  and  the  prevalence  of  the  value  under  study.  The  more  prevalent 
the  value  in  the  target  group's  value  hierarchy,  the  fewer  subjects  are 
required  to  detect  this  fact.  Conversely,  a  large  number  of  subjects 
are  needed  to  pinpoint  values  held  by  small  segments  of  the  population 
under  study. 

B.  MEASURES  OF  PREVALENCE  AND  INTENSITY 

Evidence  to  date  indicates  that  frequency  of  mention  (how  often 
a  given  value  is  cited)  reflects  the  prevalence  of  the  value  in  the 
target  population's  macro-value  structure.  In  addition,  frequency  may 
imply  intensity,  but  this  conclusion  must  be  reached  with  caution, 
because  high  frequency  can  also  be  caused  by  saliency  (i.e.,  the  par¬ 
ticular  value  stands  out  clearly  to  the  subjects)  and  "comfort"  (i.e., 
the  value  is  well-accepted  and  the  subjects  feel  comfortable  in  talking 
about  it). 

C.  TIME  FACTORS  IN  DATA  COLLECTION  PROCEDURES 

U.S.  adult  Ss  require  20  to  35  minutes  to  answer  20  ECHO  questions. 

A  group  of  third  grade  pupils  took  approximately  one  and  a  half  hours  to 
do  the  same  thing.  (An  equivalent  group  of  sixth  graders,  however,  com¬ 
pleted  the  task  in  the  same  time  as  adults.) 
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D.  HOW  MANY  VALUES  CAN  A  SINGLE  EXPERIMENT  DETECT? 

The  number  of  categories  in  a  classification  system  is  a  function 
of  the  number  of  cards,  i.e.,  the  framework  required  to  organize  the 
cards  varies  directly  with  the  number  of  cards  classified.  It  also  varies 
with  the  "set"  of  the  classifiers,  which  may  be  influenced  by  natural 
tendencies,  assumptions,  or  instructions. 

E.  QUANTIFYING  DIFFERENCES  IN  SOCIAL  PERCEPTION 

Differences  in  how  two  populations  view  the  same  behavior  can  be 
detected  and  quantified  by  having  a  single  batch  of  data  classified  by 
Cs  from  both  groups. 

F.  REDEFINING  "VALUE  DEFINITIONS" 

Meanings  assigned  by  a  group  of  indigenous  Cs  can  be  redefined 
by  having  several  other  groups  of  Cs,  from  the  same  population,  classify 
the  data. 

G.  IMPORTANCE  OF  A  PARTICULAR  VALUE  IN  THE  S's  VALUE  HIERARCHY 
The  opinions  of  individual  Ss  about  the  importance  of  a  value 

may  show  wide  variability.  .Ss'  rankings  of  item  "importance"  (I)  are 
directly  related  to  the  sequence  of  response;  (I)  values  are  statisti¬ 
cally  independent  of  categories,  i.e.,  importance  rankings  are  evenly 
distributed  among  all  categories. 

H.  DIFFERENTIATING  BETWEEN  POPULATIONS  AND  SUB-POPULATIONS 

Value  hierarchies  discovered  by  ECHO  differentiate  between  different 
populations  (eg.,  Cubans  and  U.S.  college  students)  and  between  sub¬ 
populations  in  a  single  culture  (e.g.,  males  and  females). 

I.  VALUE  OF  INDIGENOUS  CLASSIFIERS 

Experiments  in  which  teams  from  different  populations  classified 
data  from  a  single  population  demonstrated  the  value  of  indigenous 
classifiers;  these  individuals  are  able  to  impart  subtle  meanings  and 
differentiation  to  the  data  that  "experts"  often  overlook. 
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J.  CLASSIFICATION  PROCEDURES 

The  current  operational  procedure  for  categorizing  the  data 
requires  that  (1)  each  data  pool  be  classified  by  two  different  teams 
of  Cs;  (2)  the  categories  be  ranked  on  an  "importance  scale"  by  the 
_Cs,  and  (3)  the  cards  be  sorted  into  a  predetermined  set  of  categories 
as  a  test  of  classifier  reliability. 

The  data  are  divided  into  positive  and  negative  (i.e.,  good  and 
bad  thing  to  do)  samples,  and  classified  as  separate  samples  rather  than 
as  a  single  "mixed"  sample.  Categories  from  a  single-sample  classifi¬ 
cation  system  are  preferred  in  validity  tests  over  those  from  "mixed" 
classification  systems. 

Data  from  20  to  30  subjects  (200-300  cards)  can  be  classified  by 
one  team  at  one  sitting  before  fatigue  and  boredom  introduce  added 
variance  to  the  process.  Classifier  reliability,  i.e.,  the  agreement 

between  two  teams  of  indigenous  classifiers,  is  high  under  the  conditions 
described  above. 

K.  INDICATORS  OF  CLASSIFIER  EFFICIENCY 

Several  potential  indicators  of  classifier  efficiency  are  under 
consideration.  Identifying  and  training  good  indigenous  £s  should 
greatly  improve  the  efficiency  of  the  data  analysis  procedure. 

L.  ABILITY  OF  ECHO  TO  OPERATE  IN  NON-ENGLISH-SPEAKING  CULTURES 

The  ECHO  methodology,  as  we  know  it  now,  is  capable  of  operating 
effectively  in  literate  non-English-speaking,  Occidental  populations 
(e.g.,  Spanish/Cuban).  The  ability  of  the  method  to  generalize  beyond 
Judeo-Christian  cultures  to  other  cultures  (e.g.,  Thai)  will  be  evalua¬ 
ted  in  Phase  II.  The  structure  of  Oriental  languages  and  the  impact  of 
a  different  religious  philosophy  may  require  a  modification  of  the  ECHO 
methodology. 
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K.  OPERATIONAL  PROCEDURES  FOR  ILLITERATE  SUBJECTS 

ECBj  inputs  can  be  collected  from  nonhostile  illiterate  subjects 
by  oral  interviews.  However,  the  data  collection  interview  can  be  an 
extremely  time-consuming  procedure.  For  example,  subjects  have  taken  an 
hour  and  20  minutes  to  give  10  "good"  and  10  "bad"  responses.  This  seems 
to  be  a  function,  not  of  difficulty,  but  of  the  subject's  desire  to  talk 
to  someone  who  will  listen. 

Several  methods  of  gathering  and  classifying  oral  interview  data 
are  under  consideration. 

N.  WHAT  IS  THE  MINIMUM  AGE  OF  SUBJECTS  WHO  CAN  USE  THE  WRITTEN 
INSTRUMENTS? 

School  children  as  young  as  8  years  old  are  capable  of  handling 
the  IBM  card  form  of  the  ECHO  instrument;  however,  they  require  1  to 
1-1/2  hours  to  complete  20  cards. 

O.  SENSITIVITY  TO  DYNAMIC  CHANGES 

The  ECHO  methodology  appears  to  be  capable  of  detecting  dynamic 
changes  in  the  macro-value  structure  of  populations.  This  will  provide 
a  sensitive  instrument  for  tracing  the  internalization  of  particular 
values  and  the  socialization  process  in  a  culture. 

P.  "POWER  STRUCTURE"  ANALYSIS 

Perceived  sources  of  positive  and  negative  reinforcement  operating 
on  a  population  (the  "power  structure")  can  be  identified  and  quantita¬ 
tively  described  by  the  current  ECHO  analysis  system. 
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XI.  IMPLICATIONS  AND  FURTHER  STUDIES 

The  project  to  date  has  been  designed  to  perfect  the  methodology 
and  to  determine  its  feasibility.  The  results  have  seemed  impressive, 
but  the  number  of  subjects  on  some  of  the  critical  tests  has  been 
small.  Should  ECHO  be  used  in  the  field,  the  stakes  may  be  very  high; 
prudence  dictates  that  new  tests  be  made  and  the  old  ones  replicated. 

The  following  topics  are  among  those  which  must,  or  might,  be  given 
consideration  in  future  research  efforts. 

A.  MODIFYING  THE  METHODOLOGY  FOR  USE  WITH  ILLITERATE  POPULATIONS 

All  subjects  to  date  (with  the  exception  of  those  participating 
in  the  East  Los  Angeles  pilot  study)  were  literate  in  English  or 
Spanish;  applying  the  ECHO  technique  to  illiterate  populations  will 
require  some  major  modifications  in  the  methodology.  These  modifica¬ 
tions  seem  warranted  in  view  of  the  large  proportion  of  the  potential 
subject  populations  which  is  functionally  illiterate.  Several  questions 
must  be  answered:  How  can  data  be  collected  from  illiterates?  What 
system  of  data  classification  can  be  developed  to  make  use  of  indigenous 
classifiers  and  the  subtle  meanings  they  can  impart  to  the  final  classi¬ 
fication  system?  How  can  tentative  findings  be  fed  back  to  the  popula¬ 
tion  for  validation?  Several  potential  solutions  to  these  problems 
immediately  suggest  themselves.  Data,  for  instance,  could  be  collected 
in  oral  interviews,  but  what  influence  will  loss  of  subject  anonymity 
have  on  the  type  of  values  the  Ss  mention? 

For  every  potential  modification  of  the  current  methodology  there 
is  an  effect;  experiments  must  be  undertaken  to  discover  the  optimal 
way  (given  certain  economic,  efficiency,  and  social  criteria)  of  collect¬ 
ing  and  classifying  meaningful  data  from  functionally  illiterate  sub¬ 
jects. 


B.  UTILIZING  "SECONDARY"  DATA 

The  current  ECHO  methodology  makes  use  of  approximately  60  to  70 
percent  of  all  the  data  collected,  i.e.,  about  60  to  70  percent  of  the 
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data  falls  in  the  five  most  frequently  mentioned  categories  of  each 
system.  The  remaining  data  probably  hold  much  significant  information 
particularly  about  small  subcultures,  fringe  groups,  and  socially 
unpopular  or  forbidden  behaviors.  In  addition,  the  individual  responses 
within  the  five  major  categories  may  yield  important  information. 


C.  VARIATIONS  OF  THE  PROJECTIVE  SURVEY  QUESTIONNAIRE 

The  ECHO  technique  is  not  limited  to  a  single  form  of  the  pro¬ 
jective  survey  question;  the  principle  underlying  the  technique  can 
be  used  in  many  different  ways  to  discover  many  different  types  of  data. 

The  majority  of  our  experience  has  been  with  the  question:  "What 
is  a  good/bad  thing  to  do  and  who  would  approve/disapprove?"  Some 
preliminary  work  has  been  done  with  the  question:  "What  is  a  good/bad 
thing  that  could  happen  and  what  (or  who)  would  be  chiefly  responsible 
for  its  happening?"  The  first  question  seeks  to  identify  values  held  by 
the  target  population,  while  the  latter  attempts  to  tap  the  population's 
expectations  and  perceptions  of  causation  (internal  or  external  causes) . 
Several  variations  of  the  ECHO  question  have  been  formulated,  and 
research  will  be  needed  to  evaluate  their  effectiveness  in  achieving 
their  intended  aims  (e.g.,  detection  of  areas  of  conflict  between  an 
individual  and  his  society  or  sources  of  conflict  between  small  sub¬ 
cultures  and  the  larger  society) . 

D.  USE  OF  CHILDREN  AS  INDICATORS  OF  PARENTS'  CULTURE 

Phase  I  pilot  studies  indicate  that  children  eight  years  old  and 
older  are  capable  of  handling  the  standard  IBM  form  of  the  questionnaire; 
however,  little  is  known  about  their  ability  to  classify  the  data. 
Experiments  to  ascertain  this  information  should  be  initiated.  By  break¬ 
ing  the  classification  process  down  into  subtasks,  and  using  children  as 
Cs,  one  should  be  able  to  identify  the  potential  sources  of  difficulty 
when  the  classifiers  are  functionally  illiterate  or  minimally  literate. 
Research  with  children  is  preferable,  in  some  cases,  to  experiments  with 
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adult  Ss:  children  are  reliable  (e.g.,  not  one  single  child  1  failed 
to  complete  all  20  cards;  a  number  of  Ss  in  every  other  population  did 
leave  several  cards  blank),  children  are  less  suspicious  and  more 
cooperative,  and  test  populations  of  children  are  usually  available  over 
a  long  period  of  time  (e.g.,  one  semester,  one  school  year).  ECHO  data 
collected  from  children  also  appear  to  be  useful  as  indices  of  the 
culture  s  social  control  system  and  the  value  system  of  the  parents' 
society.  Research  should  be  undertaken  to  discover  (1)  the  extent  to 
which  data  can  be  collected  and  classified  by  children  and  (2)  the 
nature  of  the  linkage  between  children's  perceptions  of  their  parents' 
society  and  reality. 

E.  CREATION  OF  MESSAGES  FROM  ECHO  OUTPUTS 

The  value  hierarchies  produced  by  the  classifiers  are  independent 
declarative  sentences:  "it  is  a  good  thing  to  ...."  or  "It  is  a  bad 
thing  to  ....".  No  attempt  was  made  during  Phase  I  research  to  link 
different  values  together  to  form  credible  messages;  pilot  work  with 
ECHO  messages  during  the  summer  of  1966  indicated  clearly  that  credible 
messages  can  be  generated  from  the  classifiers'  outputs.  Which  values 
go  together  to  form  credible  persuasive  communications  must  be  discovered; 
likewise,  the  relationships  between  the  source  (implicit  or  explicit) 
of  the  communication  and  the  values  that  should  be  included  in  the 
message  must  be  investigated.  Successful  completion  of  this  research 
could  produce  a  technique  for  generating  messages  which  are  superior, 
in  credibility  or  persuasiveness,  to  those  generated  by  experts.  Couch¬ 
ing  the  communications  in  the  vocabulary  and  syntax  which  are  popular 
with  the  target  population  should  increase  the  power  of  the  messages. 

The  data  needed  to  complete  this  operation  are  produced  through:  (1) 
the  actual  data  collected  from  J3s  and  (2)  the  classifiers'  definitions 
of  the  categories.  Computerized  concordance  and  parsing  programs,  when 
applied  to  ECHO  data,  have  potential  as  objective,  unbiased  sources  of 
guidance  for  writers  of  persuasive  messages. 


UNCLASSIFIED 


39 


UNCLASSIFIED 


F.  BEHAVIOR  MODIFICATION  AND  ATTITUDE  CHANGE 

Messages  produced  in  the  manner  described  above  might  be  used  to 
modify  the  behavior  of  certain  populations.  Applying  the  ECHO  outputs 
to  behavior  and  attitude  change  tasks  will  require  a  more  sophisticated 
technique. 
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APPENDIX  I 

A  DETAILED  STATEMENT  OF  HISTORY 

The  ECHO  methodology  has  been  tested  and  refined  on  several  differ¬ 
ent  subject  populations.  The  purpose  of  this  appendix  is  to  trace  the 
development  of  ECHO,  describing  the  basic  methodology  employed  with  each 
test  population  and  the  impact  which  each  study  had  upon  subsequent 
applications  of  ECHO. 

Sections  A  and  B  describe  the  pilot  work  of  Summer,  1966.  The  rest 
of  the  appendix  describes  the  project  since  January  1967. 

A.  GENERAL  RESEARCH  CORPORATION  SECRETARIES 

1.  Subjects  (Ss) 

Twenty  female  secretaries  served  as  subjects.  The  population  was 
divided  into  two  samples  of  10  £>s ,  the  first  group  nerving  in  the  data 
collection  phase  and  the  second  group  in  the  message  session. 

2.  Data  Collection  Procedures 

In  an  individual,  verbal  interview,  each  subject  was  instructed  to 
make  as  many  responses  as  possible  to  the  following  questions: 

1.  What  could  a  person  like  yourself,  a  secretary  in  an  organi¬ 
zation  like  General  Research,  do  that  would  be  a  good  thing 
to  do  and  someone  would  praise  you  for  it?  Who  would  be  the 
person  or  persons  that  would  praise  you? 

2.  What  could  a  person  like  yourself,  a  secretary  in  an  organi¬ 
zation  like  General  Research,  do  that  would  be  a  bad  thing  to 

do  and  someone  would  reprimand  you?  Who  would  be  the 
person  or  persons  that  would  reprimand  you? 

These  responses  were  recorded  in  writing  by  the  interviewers. 
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3 .  Classification  Procedure 

The  10  Ss  generated  52  responses  to  the  positive  question  (1)  and 
72  responses  to  the  negative  question  (2).  The  interviewers  classified 

the  responses  by  creating  categories  which  represented  responses  of  simi¬ 
lar  content. 

4 .  Messages 

a.  Message  Construction 

Only  those  categories  mentioned  by  at  least  half  of  the  subjects 

were  considered  in  preparing  the  two  ECHO  messages.  These  were  the  follow¬ 
ing  categories: 


Positives ; 

Personal  behavior 
Attitude  toward  work 
Attitude  toward  boss 

Negatives : 

Attitude  toward  work 
Personal  behavior 
Attitude  toward  boss 
Security 

The  interviewers  created  one  of  the  ECHO  messages  (ECHO  +)  by  linking 
together  statements  taken  directly  from  the  selected  categories.  Literary 
style  modifications  were  kept  to  a  minimum.  The  second  ECHO  message 
(ECHO  -)  was  composed  by  changing  positive  statements  into  negative  ones 
in  such  a  way  that  the  generated  message  would  hopefully  still  sound 
plausible  to  persons  outside  of  the  subject  population.  An  example: 

'Take  the  initiative;  guess  ahead  what  he  will  need  instead  of  waiting  to 
be  told"  was  changed  to:  "It  is  not  appreciated  as  much  as  many  girls 
think  for  a  secretary  to  look  for  work  on  her  own  initiative.  Don't  try 
to  anticipate  problems  and  begin  work  before  it  has  been  asked  for." 


No.  of  Ss 


No.  of  Mentions 


6 

7 

5 


16 

13 

11 


10 

6 

7 

5 


21 

16 

11 

6 
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Two  "expert"  messages  were  also  written,  one  by  the  head  secretary 
at  General  Research  and  the  other  by  a  secretary  at  the  University  of 
California  at  Santa  Barbara. 

b.  Message  Session 

The  subjects  in  the  second  group  were  shown  the  four  messages, 
unlabeled  as  to  origin,  in  individual,  verbal  interviews  and  asked  to 
answer  the  following  questions: 

1.  These  four  messages  were  written  by  four  different  persons. 
Tell  us,  judging  from  what  they  wrote,  what  type  of  person 
do  you  think  each  of  them  is? 

2.  Only  one  of  these  messages  was  written  W  a  person  with 
actual  practical  knowledge  of  secretarial  work.  Which  one 
do  you  think  it  is? 

3.  Which  writer  would  be  the  most  helpful  source  of  advice  to  a 
beginning  secretary? 

5.  Results  and  Implications 

The  results  were  encouraging,  even  though  the  small  number  of 
subjects  in  the  population,  and  the  characteristics  of  the  investigation, 
limited  considerably  the  statistical  significance  of  the  results.  However, 
the  logical  consistency  of  the  approach  and  the  realization  that  it  was 
possible  to  write  sensible  messages  on  the  basis  of  the  data  obtained 
indicated  the  feasibility  of  the  basic  idea  and  the  possibility  that  the 
method  could  be  developed  into  a  useful  tool. 

During  this  test,  it  was  realized  that  in  order  to  have  an  unbiased 
selection  of  categories  and  to  make  fall  use  of  the  information  contained 
in  the  data  it  should  be  classified  by  members  of  the  same  population. 

This  prompted  the  second  study,  which  was  made  with  students  at  University 
of  California,  Santa  Barbara  (UCSB) . 
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B.  HIGH  SCHOOL  AND  COLLEGE  STUDENTS 

1.  Subjects  (Ss) 

One  hundred  students  enrolled  In  an  introductory  sociology  course 
during  the  1966  summer  session  at  UCSB  served  as  Ss.  They  were  divided 
into  two  equal  groups  (ECHO  I  and  ECHO  I'),  each  group  consisting  of 
about  35  high  scnool  juniors  and  about  15  college  students.  Both  groups 
received  the  same  treatment  except  that  the  Echo  I’  group  was  tested  one 
week  after  the  Echo  I  group. 

2 .  Data  Collection  Procedure 

The  ECHO  I  subjects  were  divided  into  two  equal  sections,  both 
sections  receiving  identical  instructions  and  treatment  except  that  the 
order  in  which  the  questions  were  presented  was  reversed  for  the  two 
sections  to  control  for  order  effect. 


Each  was  given  ten  blue  and  ten  brown  IBM  cards.  They  were  Instructed 
to  indicate  their  academic  status  by  printing  an  "H"  ("high  school  student") 
or  an  0  (  other")  in  the  lower  left-hand  corner  of  each  card.  They  were 
asked  to  write  legibly  and,  beginning  with  the  brown  cards,  to  answer  the 
following  question  (negative  condition) : 

For  a  person  like  yourself,  in  his  everyday  life,  give 
a  specific  example  of  something  important  that  could  happen, 
that  would  be  an  unfortumate.  unfavorable  and  undesirable 
thing  to  have  happen,  and  that,  if  it  happened  to  you,  you 
would  be  unhappy . 

This  question  was  first  read  aloud,  then  written  on  the  blackboard, 
and  finally  restated  in  a  less  formal  manner. 

After  the  students  had  written  their  first  answer  to  this  question, 
they  were  instructed  to  print  (in  the  lower  right-hand  corner  of  the  card) 
who  or  what  would  be  wholly  or  mostly  responsible  for  this  event.  At 
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this  point  the  students  were  told  that  they  would  have  ten  minutes  to 
complete  as  many  of  the  brown  ("negative")  cards  as  they  could. 

After  the  students  had  worked  for  10  minutes  on  the  brown  cards, 
they  were  instructed  to  put  them  aside  and  to  take  out  the  blue  cards. 
The  same  procedure  was  employed  with  a  different  question: 


For  a  person  like  yourself,  in  his  everyday  life,  give  a 
specific  example  of  something  important  that  could  happen 
that  would  be  a  fortunate,  favorable  and  desirable  thing 
to  have  happen  and  that,  if  it  happened  to  you,  you  would 
be  pleased. 


Again,  the  students  were  instructed  to  indicate  the  "source,"  i.e.,  the 
person  who  would  be  mainly  responsible  for  this  thing  happening.  They 
were  given  10  minutes  to  complete  as  many  blue  cards  as  they  could. 


The  students  were  then  instructed  to  put  the  blue  cards  aside  and 
to  rate  the  brown  cards  in  terms  of  their  importance  using  the  "I"  box 
(upper  left-hand  corner  of  the  card).  The  instructions  were  as  follows: 

If  you  had  one  wish  and  you  could  make  sure  that  one  of 
these  things  would  never  come  true... then  write  in  the  upper 
left-hand  corner  of  the  card  the  number  1.  Now  look  at  the 
remaining  cards  and  do  the  same  thing.  The  biggest  number 
will  then  be  the  one  thing  that  you  would  allow  to  happen  if 
one  thing  had  to  happen. 

In  this  manner,  the  subjects  assigned  importance  numbers  to  first  the 
brown  cards  and  then  the  blue  cards. 


The  final  task  was  the  assignment  of  an  "uncertainty"  or  probability 
(P)  estimate  to  the  cards.  The  instructions  were  'is  follows: 

Now  take  the  brown  cards  again  and  spread  them  out.  Ignore  the 
number  you  have  put  in  the  "I"  box.  Look  at  the  answers  and, 
taking  them  in  any  order,  ask  yourself  the  following  thing 
about  the  event  you  have  specified:  "How  likely  is  this  to 
come  to  be  true?"  Think  of  it  in  this  way:  If  a  hundred  people 
like  yourself  all  had  the  desire  for  this  outcome,  how  many 
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do  you  think  would  get  their  wish?  Write  that  number  in 
the  upper  right-hand  corner  of  the  card. 

In  this  manner,  the  subjects  assigned  uncertainty  values  to  all  of  the 
events.  Similar  instructions  were  given  for  the  blue  cards. 


Each  card  was  completed  as  shown  in  Fig.  5. 


I  = 

P  = 

SPECIFIC  EVENT 

H  or  0 

SOURCE  : 

Figure  5.  Data  Collection  Card  Used  in  ECHO  I  Study 


3.  Classification  Procedure 

Three  persons  indigenous  to  the  subject  population  (i.e.,  students 
from  the  class)  served  as  classifiers  and  sorted  the  responses  in  the 
following  r.anner: 

1.  Each  classifier  was  given  one-third  of  the  blue  IBM  cards 
(positive  responses)  and  instructed  to  put  together  those 
cards  which  had  the  same  meaning. 

2.  The  first  classifier  to  complete  this  initial  sorting  made 
labels  for  the  categories  he  had  created.  When  the  others 
had  finished  their  sorting,  the  first  classifier  read  each 
of  his  labels  aloud  and  the  others  handed  him  their  cards 
which  they  believed  fell  into  the  same  category.  In  this 
manner,  the  three  separate  classification  systems  were  com¬ 
bined  to  make  one. 
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3.  Each  category  was  then  discussed  individually.  The  cards 
were  read  aloud  and  the  classifiers  were  asked  to  decide 
which  of  the  responses  actually  belonged  in  the  category. 

When  it  seemed  appropriate,  categories  were  broken  down 
into  smaller  categories.  The  resulting  classification 
system  consisted  of  15-20  categories  describing  the  subjects' 
responses  to  the  positive  question.  The  brown  cards,  i.e., 
responses  to  the  negative  question,  were  subsequently  classi¬ 
fied  in  the  same  manner  by  the  same  procedure. 

4 .  Results  and  Implications 

No  message  session  was  employed  because  this  study  was  designed 
to  examine  the  relationships  between  the  subjects'  expectations  and  the 
uncertainty  and  importance  they  attached  to  those  attitudes.  Several 
mathematical  relationships  were  proposed  but  the  ordinal  nature  of  the 
importance  and  uncertainty  measures  prohibited  useful  mathematical  manipu¬ 
lation  of  the  variables. 

However,  the  results  did  show  that,  in  principle,  a  classification 
method  employing  indigenous  classifiers  could  be  developed  in  such  a  way 
as  to  fully  use  the  information  collected  via  the  ECHO  methodology. 

In  addition,  ECHO  appeared  to  be  an  effective  tool  for  discovering 
and  analyzing  the  different  systems  of  social  control  of  the  various 
subgroups  within  the  subject  culture. 

C.  DEVELOPMENT  OF  ECHO  DATA  COLLECTION  INSTRUMENT 

1.  Materials 

IBM  cards  were  chosen  as  the  data  collection  instrument  because 
of  easy  manipulation  in  the  classification  session,  efficient  sorting 
and  later  retrieval  of  information,  and  direct  access  to  computer  opera¬ 
tions  without  transferring  data.  Data  collection  packets  were  made  up 
as  follows: 
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1.  iwenty  IBM  cards  were  presented  to  each  subject:  10  were 
printed  with  the  positive  questions:  "What  is  a  good  thing 

to  do?  Who  would  approve?"  and  10  with  the  negative  questions 
"What  is  a  bad  thing  to  do?  Who  would  disapprove?" 

2.  The  cards  were  prepunched  with  numbers  identifying  the  subject 
the  population  from  which  he  was  drawn,  the  valence  of  the 
assigned  evaluation  (positive  or  negative) ,  and  the  sequence 
in  which  the  _S  wrote  his  responses. 

3.  Ranking  instruction  cards  were  included  in  the  packets, 
instructing  the  Ss  to  rank  their  responses  to  each  question 
from  1  to  10  in  order  of  decreasing  importance. 

4.  Role  instruction  cards,  placed  first  in  the  packets,  instruc¬ 
ted  the  subjects  to  assume  specific  roles  in  answering  the 
ECHO  questions. 

5.  Biographical  information  cards,  placed  last  in  the  packets, 
requested  information  about  the  _S's  age,  sex,  marital  status, 
academic  status,  and  length  of  attendance  at  the  school. 

2.  Application  of  ECHO  Data  Collection  Instrument — UCLA  Pilot  Study 

a.  Data  Collection  Conditions 

(1)  Role  Assignments 

Etch  person  may  play  several  different  roles  concurrently.  For 
example,  a  man  might  be  a  student,  a  son,  a  husband,  and  an  employee  at 
the  same  time.  He  will  have  different  systems  or  hierarchies  of  values 
and  attitudes  to  correspond  with  these  different  roles.  For  example, 
in  his  role  as  a  husband,  attitudes  and  actions  related  to  his  rela¬ 
tionship  with  his  wife  might  be  of  greatest  importance  while,  in  his  role 
as  a  student,  studying  and  getting  good  grades  will  predominate.  There¬ 
fore,  in  trying  to  assess  the  person's  value  system,  it  is  necessary  to 
define  the  context,  or  role,  in  terms  of  which  he  is  to  respond;  that 
is,  to  indicate  the  value  system  and  its  attendant  role  in  which  we  are 
interested . 
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The  UCLA  subjects  were  studied  in  terms  of  two  different  roles: 
some  of  the  subjects  were  instructed,  via  printed  role  instruction  cards 
placed  first  in  their  packets,  to  answer  the  ECHO  questions  in  terms  of 
their  roles  as  "students  at  UCLA";  others  were  given  no  specific  role 
instructions  and  hence  could  assume  any  role  they  desired.  This  second 
condition  was  termed  the  "general  person  role". 

(2)  Sequential  Conditions 

To  control  for  any  possible  sequential  effects  which  might  result 
from  the  order  in  which  the  positive  and  negative  questions  were  answered, 
the  Ss  were  divided  into  two  groups.  Group  A  answered  the  10  positive 
questions  first  and  the  10  negative  questions  second;  Group  B  the  reverse. 

(3)  Ranking  Conditions 

All  the  subjects  were  instructed  to  rank  their  responses  in  order 
of  importance.  Three  ranking  conditions  vjere  employed  in  an  effort  to 
determine  at  what  place  in  the  tests  the  subjects  should  rank  their 
responses.  The  Ss  under  ranking  condition  I  completed  the  cards  in  deck  1, 
completed  the  cards  in  deck  2,  ranked  deck  1,  and  ranked  deck  2.  Ranking 
condition  II  Ss  completed  deck  1,  ranked  deck  1,  completed  deck  2,  and 

ranked  deck  2.  For  ranking  condition  III,  positive  and  negative  cards 
were  intermixed  in  decks  1  and  2.  These  subjects  completed  deck  1, 
completed  deck  2,  ranked  deck  1,  and  ranked  deck  2.  The  way  in  which 
the  Ss  were  grouped  by  role,  sequential  condition,  and  ranking  condition, 
is  shown  in  Table  8. 

TABLE  8 


DESIGN  OF  DATA  COLLECTION  CONDITIONS— UCLA  PILOT  STUDY 


Ranking  Condition 


Role  Condition 

I 

II 

III 

Student 

A(N  =  20) 

A(N  =  20) 

N  =  17 

B  (N  =  20) 

B(N  -  20) 

General  Person 

A(N  =  20) 

' 

B(N  =  20) 
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b.  Subjects 

One  hundred  thirty-seven  students  enrolled  in  an  undergraduate  psy¬ 
chology  course  at  UCLA  served  as  subjects.  The  subjects  were  predominately 
upper  classmen  (juniors  and  seniors)  and  their  ages  ranged  from  18  to  47 
years . 


c.  Method 

(1)  Materials 

Each  subject  received  a  packet  consisting  of  20  printed,  prepunched 
IBM  cards,  2  ranking  instruction  cards,  and  a  biographical  information 
card.  The  subjects  under  the  "student  role"  condition  also  received  role 
instruction  cards. 

(2)  Procedure 

After  the  packets  had  been  distributed  and  the  instructions  read 
by  the  investigator,  the  subjects  were  permitted  to  complete  their 
packets  at  their  own  speed.  The  instructions  to  the  subjects,  which  were 
read  aloud  by  the  investigator,  were  as  follows: 

INSTRUCTIONS  TO  THE  SUBJECTS 


The  purpose  of  this  study  is  to  develop  methods  for 
improving  communications  between  different  cultures.  Any 
such  interculture  communication  is  necessarily  based  on  an 
understanding  of  the  value  systems  inherent  in  each  culture. 

e  are  currently  investigating  an  instrument  with  which  we  hope 
we  will  be  able  to  discover  and  examine  the  value  system  for 
any  cultures.  We  plan  to  test  the  effectiveness  of  our  metho¬ 
dology  by  comparing  the  culture  which  has  developed  in  this 
group  with  a  somewhat  similar  culture  which  has  developed 
within  another  group  of  students. 

Each  of  you  should  have  a  sealed  envelope  and  a  pen. 

These  are  the  only  materials  you  will  need.  You  will  be  asked 
to  give  ten  responses  to  each  of  the  following  questions:  "What 
do  you  think  a  person,  like  yourself,  could  do  that  would  be 
a  good^thing  to  do  and  that  someone  would  approve  of  your 
doing?"  "Who  would  approve  of  your  action?"  Likewise  "What 
do  you  think  a  person,  like  yourself,  could  do  that  would  be  a 
bad  thing  to  do  and  that  someone  would  disapprove  of  your 
doing  ?  "Who  would  disapprove?" 
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When  we  tell  you  to  begin,  you  are  to  open  the  envelope 
by  tapping  it  down  on  the  left  side  so  that  the  cards  within 
are  down  on  the  left.  Then  tear  off  the  right  edge.  Take  out 
the  deck  of  cards  which  is. numbered  with  a  "1".  Do  not  remove 
anything  else  from  the  envelope.  Take  the  rubber  band  off  the 
cards  but  be  especially  careful  that  you  do  not  change  the 
order  of  the  cards.  Writing  or  printing  as  neatly  as  possible 
give  a  specific  example  of  a  good  or  bad  thing  to  do,  accord¬ 
ing  to  the  question  which  is  printed  on  the  card.  Write  that 
answer  on  the  lines  which  have  been  provided  following  the 
question.  Then  write  the  title  or  position  of  the  person  who 
would  approve  or  disapprove  of  your  action  on  the  lines  which 
have  been  provided  following  that  question.  When  you  have 
answered  all  ten  cards  in  deck  1,  put  them  back  into  the 
envelope  and  take  out  deck  "2".  Again,  we  ask  you  not  to  change 
the  order  of  the  cards.  Follow  the  same  procedure  as  before 
in  writing  your  answers  on  these  cards.  When  you  are  finished, 
return  these  cards  to  the  envelope  and  take  out  the  yellow 
"Biographical  Information"  card.  Answer  each  of  the  questions 
on  this  card  by  placing  an  "x"  in  the  appropriate  box.  We 
wish  to  thank  you  all  again  for  donating  your  time  to  help  make 
this  study  a  success. 

Do  you  have  any  questions?  If  a  problem  should  arise  while 
you  are  working,  please  raise  your  hand.  You  may  begin. 


In  response  to  requests  for  clarification,  the  investigator  gave 
the  Ss  the  following  example:  "If  a  fireman  were  asked  'What  is  a  good 
thing  to  do?'  and  'Who  would  approve?',  he  might  answer,  'A  good  thing  to 
do  is  to  keep  my  fire  engine  in  good  running  condition  and  the  fire  chief 
would  approve  of  this  act."'  In  addition,  the  subjects  asked  whether  or 
not  they  could  list  themselves  as  sources  of  approval  or  disapproval. 

The  investigator  answered  that  they  could  indeed  list  themselves  or  any¬ 
one  else  as  sources. 


3 •  Results  of  UCLA  Pilot  Study  of  Data  Collection  Instrument 

a.  Evaluation  of  Data  Collection  Instrument 

The  data  collection  packet,  consisting  of  printed,  prepunched  IBM 
cards,  is  an  efficient  means  of  collecting  responses  to  the  ECHO  questions. 
The  Ss  easily  handled  the  20  cards,  responding  to  all  of  them  in  20-30 
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minutes.  No  sequential  effect  arising  from  the  order  of  presentation  of 
positive  and  negative  evaluations  was  discovered. 

However,  an  ordinal  effect  was  found  rel-iting  the  order  in  which 
specific  responses  were  written  and  the  importance  estimates  assigned  by 
the  subject  to  those  responses.  In  general,  the  subjects  tended  to  write 
the  most  important  responses  first.  See  Appendix  II. 

The  instructions  to  the  subjects  proved  to  be  adequate  in  that 
almost  all  subjects  correctly  followed  the  prescribed  procedure.  How¬ 
ever,  two  changes  were  made  in  the  instruction  for  future  studies. 


1.  No  examples  of  the  desired  responses  will  be  given  as  any 
such  example  might  bias  the  Ss '  responses. 

2.  The  instructions  will  indicate  that  the  Ss  may  list  themselves 
as  sources  of  approval  or  disapproval  if  they  wish. 


b.  Data  Collection  Sessions  at  Carnation  and  Prudential 
(1)  Subjects 

These  two  groups,  studied  concurrently,  consisted  of  clerical  employ¬ 
ees.  The  compositions  of  the  two  groups  differed  slightly.  The  54  Carna¬ 
tion  subjects  (39  secretaries  and  15  clerk-typists)  on  the  average,  were 
older  and  had  more  education  than  the  Prudential  subjects.  In  addition, 
they  were  more  homogeneous,  in  terms  of  age,  education,  and  length  of 
employment,  than  the  Prudential  subjects.  The  Prudential  subject  population 
consisted  of  women  employed  at  Company  job  levels  3,  4  and  5,  the  majority 
working  as  clerks.  Of  this  group,  93  were  chosen  randomly  to  serve  in  the 
data  collection  phase  of  the  study. 
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(2)  Data  Collection  Procedure 

(a)  Materials 

Each  data  collection  packet  consisted  of  20  IBM  cards.  Ten  of  the 
cards  had  printed  on  them:  "What  is  a  good  thing  to  do?"  "Who  would 
approve?"  and  the  other  ten  cards  read:  "What  is  a  bad  thing  to  do?" 

"Who  would  disapprove?"  Each  card  was  prepunched  with  group  and  subject 
identification  numbers  and  a  number  to  identify  the  sequence  in  which  the 
cards  were  filled  out.  Ranking  instruction  and  biographical  information 
cards  were  also  included  in  the  packets.  The  former  instructed  the 
subject  to  rank  the  cards  in  each  deck  in  the  order  of  decreasing  impor¬ 
tance.  The  latter  elicited  information  regarding  the  subject's  age, 
sex,  education,  length  of  emt- 1  oyment ,  and  job  title. 

(b)  Procedure 

The  packets  were  distributed  and  the  subjects  instructed  to  work 
at  their  own  speed  in  answering  the  questions  printed  on  the  cards. 

All  the  subjects,  with  the  exception  of  12  Carnation  subjects, 
were  instructed  to  list  good  and  bad  things  to  do  in  terms  of  their  roles 
as  employees.  The  12  Carnation  subjects  were  instructed  verbally  to  ans¬ 
wer  the  questions  in  terms  of  their  roles  as  persons,  i.e.,  in  terms  of 
their  roles  in  private  life,  rather  than  restricting  themselves  to  work- 
oriented  responses. 

The  following  instructions  were  read  to  the  subjects. 

INSTRUCTIONS  TO  THE  SUBJECTS 


Before  we  begin,  we  want  to  assure  you  that  your  answers 
will  be  considered  confidential  and  that  no  company  personnel 
will  have  access  to  them.  In  addition,  your  anonymity  is 
assured  as  there  is  no  possible  way  in  which  we  could  identify 
any  of  you  from  your  answers.  We  wish  to  emphasize  that  these 
questions  are  not  designed  as  tests  of  your  ability  or  intelli¬ 
gence.  Also,  there  are  no  right  or  wrong  answers  as  we  merely 
want  your  opinion. 
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Each  of  you  has  a  sealed  envelope  and  something  to  write 
with.  These  are  the  only  materials  you  will  need  this  evening. 

You  will  be  required  to  list  ten  responses  to  each  of  the 
following  questions:  These  questions  are:  "What  do  you  think  a 
person  like  yourself,  in  your  position  at  this  company,  could 
do  that  would  be  a  good  thing  to  do  and  that  someone  would 
approve  of  your  doing?"  and  "What  is  the  job  title  or  position 
(NOT  THE  NAME)  of  the  person  who  would  approve  of  your  doing 
that  thing?"  By  this,  we  mean — who  would  be  aware  that  you  had 
done  this  and  who  would  be  "pleased"?  "What  do  you  think  a 
person  like  yourself  in  your  position  at  this  company  could  do 
that  would  be  a  bad  thing  to  do  and  that  someone  would  disapprove 
of  your  doing?"  and  "What  is  the  job  title  or  position  of  the 
person  who  would  disapprove  of  your  doing  that  thing?"  vor  ex¬ 
ample,  the  person  who  approves  or  disapproves  might  be  your  friends, 
your  boss,  employer,  supervisor,  your  mother,  or  yourself. 


Open  the  envelope  by  tapping  it  down  on  the  left  side  so 
that  the  cards  within  it  are  down  on  the  left.  Then  tear  off 
the  right  edge.  Take  out  the  deck  of  cards  which  is  numbered 
with  a  "1".  Do  not  remove  anything  else  from  the  envelope. 

Take  the  rubber  band  off  the  cards  but  be  especially  careful 
that  you  do  not  change  the  order  of  the  cards.  Writing  or 
printing  as  neatly  as  possible  give  a  specific  example  of  a 
good  thing  or  a  bad  thing  to  do,  according  to  the  question  which 
is  printed  on  the  card.  Write  that  answer  on  the  lines  which 
have  been  provided  following  the  question.  Then  write  the  title 
or  position  of  the  person  who  would  approve  or  disapprove  of  your 
action  on  the  lines  which  have  been  provided  following  that 
question.  When  you  have  answered  all  ten  '■'ards  in  deck  1,  put 
the  rubber  band  back  around  the  cards,  making  sure  that  the 
piece  of  paper  with  the  large  number  "1"  is  attached  to  them, 
and  put  the  cards  aside.  Then  take  the  deck  of  cards  with  the 
large  number  "2"  out  of  the  envelope  and  remove  the  rubber  band. 
Again,  we  ask  you  not  to  change  the  order  of  the  cards.  Follow 
the  same  procedure  as  before  in  writing  your  answers  on  these 
cards.  When  you  are  done,  replace  the  rubber  band  and  the  "2" 
and  put  the  cards  aside.  When  you  have  finished  answering  the 
cards  in  deck  "2"  you  will  be  ready  to  begin  the  next  step  in 
this  survey.  You  will  find  two  white  cards  in  your  envelope. 
These  cards  will  instruct  you  to  rank  your  answers  in  terms  of 
how  important  each  answer  is  to  you.  Rank  deck  "1"  first. 

You  can  do  this  in  the  following  manner.  Spread  the  cards  out 
on  the  table  in  front  of  you  so  that  you  can  read  the  ten 
actions  you  wrote.  Pick  the  action  which  you  think  is  the 
most  important  and  put  a  "1"  in  the  box  in  the  lower  left- 
hand  corner  of  the  card.  Now,  find  the  action  which  is  second 
in  importance  to  you  and  put  a  "2"  in  the  box. 
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wrote  the  PU  uf  u3  ln  the  boX  °n  the  card  on  which  you 

' the/ ac‘*°”  whlch  y°u  think  is  third  in  importance. 

inue  in  this  manner  until  you  have  ranked  all  ten  cards. 

^e?ny°^!re  y°U  WiU  have  nunit)ered  the  cards  from  1 

10  with  a  1  on  the  most  important  card  and  a  ”10"  on  the 
least  important  card.  Please  recheck  the  cards  to  make  sure 
J“\cards  have  the  same  number  on  them.  When  you  are 

larve  rePlace  tbe  rubber  band  and  paper  numbered  with  the 
large  1  and  put  these  cards  aside.  Then,  you  are  to  rank 
the  ten  cards  in  deck  "2”  in  the  same  manner!  After  yo" 
have  ranked  both  decks  of  cards  take  the  yellow  card  out  of 
the  envelope.  Answer  each  of  the  questions  on  the  Biographi- 

boi  Sr3?  rh^^  ^  PlaClng  3n  ,'X"  in  the  aPPropriate 

are'f^ch  a  fi1  Part  °f  tond8ht ' s  session.  When  you 

hrfnf  ^  PUt  3  1  the  Cards  back  in  the  envelope  and 

bring  them  down  to  the  front  of  the  room. 

.  . ,D°  you  have  any  questions  at  this  time?  If  any  problems 

You^/Jeeln  ^  “*  ""“"S’  Pl“"  ralSe  y°ur  ha"d- 


D.  DEVELOPMENT  OF  DATA  CLASSIFICATION  METHOD — UCLA  PILOT  STUDY 
1*  Classification  Teams 

Two  teams  of  three  persons  each  were  chosen  from  the  subject  popula¬ 
tion  to  serve  as  indigenous  classifiers.  One  team  consisted  of  two  males 

and  one  female  while  the  other  team  was  made  up  of  two  females  and  one 
male. 


2.  Procedure 


a.  Initial  Sorting 

team  °f  three  classifiers  received  1143  positive  cards  while  the 
other  team  received  1088  negative  cards. 

Each  classifier  (C)  was  given  one-third  of  the  cards  to  be  classi¬ 
fied  by  his  team  and  was  instructed  to  read  all  the  cards  and,  then,  to 
put  together  those  cards  which  had  the  same  meaning.  This  initial  sorting 
resulted  in  several  small  stacks  of  cards  representing  different  types 
of  responses  or  categories. 
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b.  Creation  of  Larger  Categories 

The  members  of  the  classification  teams  worked  together  in  this 
phase  of  the  session.  Taking  turns,  each  C  picked  up  one  of  his  stacks 
of  cards  and  read  the  cards.  The  other  Cs  then  handed  him  all  of  their 
cards  which  they  felt  belonged  in  the  category  under  discussion.  In  this 
manner  the  three  separate  sets  of  categories  created  in  the  initial 
sorting  were  combined  to  form  one  classification  system. 


c.  Refinement  of  Categories 

The  Cs  discussed  each  individual  category,  reading  all  of  the 
cards,  and  occasionally  divided  large  categories  into  smaller  ones. 

d.  Assignment  of  Category  Labels 

The  Cs  labeled  the  categories,  creating  titles  which,  in  the  form 
of  a  response  to  the  original  ECHO  question,  would  represent  all  of  the 
responses  wi^.jin  each  category. 

e.  Estimation  of  Category  Importance 

The  C's  final  task  was  to  rank  the  categories  from  1  to  N,  where 
N  equals  the  number  of  categories,  in  order  of  decreasing  importance. 

3>  Evaluation  of  Data  Classif ication— UCLA  Pilot  Study 

The  method  itself  was  workable  and  seemed  logical  in  that  the  classi¬ 
fication  progresses  from  the  creation  of  small,  exclusive  stacks  of  cards 
to  larger,  more  general  categories. 

The  use  of  three  classifiers  per  team  appeared  to  be  an  efficient 
arrangement.  Small-groups  research  has  proven  that  triads  are  superior 
to  dyads  in  making  decisions  since  there  is  a  majority  when  an  odd  number 
of  persons  are  employed.  Three  also  seems  a  good  number  as  any  more,  say 
five,  would  have  made  the  group  too  big,  resulting  ir  less  direct  inter¬ 
action  among  the  members . 
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The  classification  session,  lasting  roughly  12  hours  because  of 
the  large  number  of  cards,  was  clearly  too  long.  The  team  which  sorted 
the  positive  responses  created  69  categories  from  the  1143  cards  while 
the  other  team  created  71  categories  from  the  1088  negative  cards. 

The  length  of  the  session,  the  enormousness  of  the  task  itself,  and 
the  lack  of  regular  breaks  led  to  the  development  of  extreme  fatigue  and 
boredom  among  the  classifiers.  Therefore,  the  reliability  of  the  result¬ 
ant  classification  systems  was  questioned. 


In  addition,  the  classifiers  created  such  a  large  number  of  cate¬ 
gories  that  the  categories  themselves  represented  specific  actions  rather 
than  generalized  attitudes  and  hence,  were  too  specific  and  detailed  to 
represent  the  more  generalized  attitudes  of  the  subject  population.  The 
multiple  classification  process  discussed  later  would  have  resolved  this 
problem.  No  conclusions  were  reached  on  the  efficacy  of  classifying 
positive  and  negative  responses  together  or  separately. 


4’  Data  Classification  Sessions  at  Carnation  and 

a*  Classification  Teams 

Two  teams  (CA  and  CB)  of  three  Cs  each  were  employed  to  classify  the 
1042  responses  from  Carnation.  Two  teams  ^  and  ?3>  of  3  Cs  and  one 
team  ( P 2>  of  2  Cs,  due  to  the  unexpected  absence  of  one  classifier,  were 
employed  to  sort  the  1671  cards  from  Prudential.  The  classification 
sessions  for  the  two  different  subject  groups  were  held  concurrently. 

b.  Distribution  of  Cards 

Carnation.  Classification  team  CA  received  516  positive  and 

negative  cards  while  team  CB  classified  526  positive  and  negative 
cards . 

Prudential.  Classification  tear,  Px  sorted  599  positive  cards; 

team  P2  sorted  472  negative  cards;  and  team  P3  sorted  600  positive 
and  negative  cards. 
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c.  Procedure 

(1)  Initial  Sorting 

As  in  the  UCLA  classification  session,  the  classification  team  mem¬ 
bers  worked  independently  in  their  initial  sorting  of  the  cards. 

(2)  Creation  of  Larger  Categories 

The  team  members  were  brought  together  for  this  phase  in  which 
the  individual  classifiers'  stacks  of  cards  were  combined  to  form  larger 
categories . 

The  investigators  circulated  among  the  various  classification  teams 
to  prevent  any  one  member  from  dominating  his  team's  discussions  and  to 
keep  the  session  moving  smoothly. 

(3)  Refinement  of  Categories 

Taking  turns,  the  Cs  read  all  of  the  cards  in  each  category.  Where 
they  felt  it  was  necessary,  the  Cs  refined  a  category  by  omitting  or 
adding  cards  or  breaking  it  down  into  two  smaller  categories.  The  final 
classification  systems  contained  the  following  numbers  of  categories: 
Carnation — team  CA  created  28  categories  and  team  CB  created  30;  Pruden¬ 
tial — team  P^  created  22  categories,  team  created  29  categories,  and 
team  P^  created  20  categories. 

(4)  Assignment  of  Category  Labels 

The  Cs  labeled  the  categories,  creating  titles  which,  in  the  form 
of  a  response  to  the  original  ECHO  question,  represented  all  of  the 
responses  within  each  category.  As  these  labels  were  general,  the  Cs 
also  listed  actions  and  attitudes  as  examples  of  the  behavior  represented 
by  the  category  titles. 

(5)  Estimation  of  Category  Importance 

The  Cs  ranked  their  categories  from  1  to  N  (where  N  equals  the 
total  number  of  categories)  in  order  of  decreasing  importance.  As  a  test 
of  importance  ranking  reliability,  Carnation's  team  CA  ranked  by  impor¬ 
tance  the  categories  created  by  team  CB.  The  Spearman's  Rank-Dif ference 
correlation  between  the  two  rankings  of  the  CB  categories  resulted  in  a 
significant  Spearman  rho  ■  .79,  a  ■  .01*. 
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Both  teams  of  Carnation  classifiers  ranked  as  most  important 
the  categories  created  from  responses  of  12  Carnation  data  collection 
subjects  who  were  assigned  "general  person  roles"  rather  than  being 
restricted,  as  were  all  of  the  other  Ss,  to  work-oriented  responses. 

This  result  led  to  some  discussion  regarding  the  assignment  of  roles. 

It  was  felt  that,  perhaps,  the  most  important  attitudes  would  be  elicited 
if  the  Ss  were  allowed  to  assume  the  roles  which  they,  themselves,  con¬ 
sidered  the  most  important  parts  of  their  lives. 

E.  APPLICATION  OF  ECHO  METHODOLOGY  AT  STANFORD 

1«  Subjects 

Seventy-one  men  and  1  woman  enrolled  in  the  Graduate  School  of 
Business  at  Stanford  University  served  as  subjects.  The  subjects  ranged 
in  age  from  22  to  40  years. 

^ •  Data  Collection  Procedure 

a.  Materials 

Data  collection  packets  identical  to  those  employed  in  the  UCLA 

pilot  study  were  prepared. 

b.  Data  Collection  Conditions 

(1)  Role  Assignments 

Half  of  the  Ss  were  instructed  by  role  instruction  cards  to  assume 
roles  as  "students  at  Stanford"  in  completing  their  packets.  The  others 
were  instructed  to  assume  their  more  general  roles  as  "persons  in  society". 

(2)  Sequential  Conditions 

Group  A  Ss  received  the  10  positive  cards  first  and  the  10  negative 
cards  last.  The  reverse  was  true  for  the  Group  B  Ss.  The  distribution 

among  the  Ss  of  the  role  assignments  and  sequential  conditions  is  shown 
in  Table  9. 
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(3)  Ranking  Conditions 

All  the  Ss  were  instructed  to  write  deck  1,  rank  deck  1,  write 
deck  2,  and  rank  deck  2.  (This  is  identical  to  ranking  condition  II 
employed  in  the  UCLA  study) .  The  use  of  this  sequence  was  based  on  the 
assumption  that  the  _Ss  would  be  best  able  to  compare  and  rank  their 
responses  immediately  after  writing  them. 


TABLE  9 

DESIGN  OF  DATA  COLLECTION  CONDITIONS— STANFORD 


Sequential  Control 


Role  Condition 

A(+,  -) 

B(- ,  +) 

Total 

Student  Role 

N  =  18 

N  -  18 

36 

General  Person 

N  -  18 

Z 

n 

00 

36 

Total 

36 

36 

72 

c.  Procedure 

After  the  data  collection  packets  had  been  distributed,  the  instruc¬ 
tions  were  read  aloud  to  the  subjects. 

INSTRUCTIONS  TO  THE  SUBJECTS 


The  purpose  of  this  study  is  to  develop  methods  for 
improving  communications  between  different  cultures.  Any  such 
inter-culture  communication  is  necessarily  based  on  an  under¬ 
standing  of  the  value  systems  inherent  in  each  culture.  We 
are  currently  investigating  an  instrument  with  which  we  hope 
we  will  be  able  to  discover  and  examine  the  value  system  for 
any  culture.  We  plan  to  test  the  effectiveness  of  our  metho¬ 
dology  by  comparing  the  culture  which  has  developed  in  this 
class  with  a  somewhat  similar  culture  which  has  developed 
within  a  class  of  students  at  UCLA. 

Each  of  you  should  have  a  sealed  envelope  and  a  pen. 
These  are  the  only  materials  you  will  need.  You  will  be  asked 
to  give  ten  responses  to  each  of  the  following  questions: 
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"What  do  you  think  a  parson,  like  yourself,  could  do  that  would 
be  a  good  thing  to  do  and  that  someone  would  approve  of  your 
doing?"  "What  is  the  position,  with  respect  to  yourself,  of 
the  person  who  would  approve  of  your  action?"  Likewise,  "What 
do  you  think  a  person,  like  yourself,  could  do  that  would  be 
a  bad  thing  to  do  and  that  someone  would  disapprove  of  your 
doing?"  "What  is  the  position,  with  respect  to  yourself,  of 
the  person  who  would  disapprove?" 

You  will  write  your  responses  on  the  IBM  cards  which  are 
in  the  envelope.  You  are  to  open  the  envelope  by  tapping  it 
down  so  that  the  cards  within  it  are  down  at  the  left  end  of 
the  envelope.  Then  tear  off  the  r.lght  edge  of  the  envelope, 
forming  a  packet  for  the  cards. 

After  you  have  read  the  special  instruction  card  which 
is  the  first  card  in  the  packet,  take  out  the  deck  of  cards 
which  has  been  marked  with  e  number  "1".  Be  especially  care¬ 
ful  :hat  you  do  not  disturb  the  order  of  these  cards  while 
you  answer  the  questions  on  them.  When  you  have  answered  all 
ten  cards,  follow  the  instructions  at  the  back  of  deck  "1". 
After  you  have  finished,  replace  the  rubber  band  around  deck 
"1"  and  go  on  to  deck  "2",  following  the  same  procedure. 

Upon  completion  of  deck  "2",  f i 11  out  the  biographical 
data  card  and  then  put  all  the  materials  back  into  the  packet. 

Bring  the  packet  to  the  front  of  the  room  and  hand  it  to 
one  of  us  as  you  leave.  We  wish  to  thank  you  for  vour  coopera¬ 
tion.  Are  there  any  questions?  You  may  begin. 


3 .  Data  Classification  Session 


a.  Classification  Teams 

Two  classification  teams,  each  consisting  of  three  men  selected 
from  the  subject  population,  were  employed  in  the  classification  session. 


b.  Distribution  of  Cards 

One  team  sorted  the  710  positive  responses  while  the  other  team 
sorted  the  695  negative  responses. 


C.  Procedure 

The  data  classification  procedure  was  identical  to  that  employed 
at  UCLA,  progressing  from  the  initial  sorting  to  the  creation  of  larger 
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categories  to  the  refinement  and  labeling  of  the  categories  and,  finally, 
to  the  ranking  of  the  categories  in  terms  of  importance.  The  resultant 
classification  systems  consisted  of  29  positive  categories  and  31  negative 
categories . 

F.  DEVELOPMENT  OF  A  VALIDITY  TEST— UCLA  PILOT  STUDY 


1.  Purpose  of  Validity  Test 

The  validity  test,  or  "message  session",  was  designed  as  an  evalua¬ 
tion  of  the  effectiveness  of  the  ECHO  data  collection  instrument  in  deter¬ 
mining  the  prevalent  and  important  attitudes  within  the  subject  culture's 
value  system.  Subjects  were  instructed  to  compare  and  evaluate  actions 
and  attitudes  represented  by  lists  of  category  titles  taken  from  the 
data.  Their  reactions  to  these  lists  indicated  the  degree  to  which  the 
lists,  and  hence  the  original  data,  actually  represented  the  actions  and 
attitudes  which  were  important  to  them. 

2.  Construction  of  Lists 

A  "message"  consisted  of  a  list  of  category  titles  from  one  classi¬ 
fication  session.  In  this  study,  the  titles  were  selected  on  the  basis 
of  category  frequency,  i.e.,  the  total  number  of  responses  in  each  category. 

The  original  UCLA  positive  categories  were  ranked  by  frequency  and 
two  lists  were  constructed  based  on  those  rankings:  Message  I  included 
the  five  most  frequent  categories  (ranks  1-5)  and  Message  II  the  second 
five  most  frequent  categories  (ranks  6-10) .  The  original  Stanford  posi¬ 
tive  categories  were  also  ranked  by  frequency  and  two  messages  (III  and 
IV)  constructed  in  a  similar  fashion.  Lists  were  also  based  on  the  five 
most  frequent  negative  categories  from  both  the  Stanford  and  UCLA  subject 
populations . 

3.  Presentation  of  Lists 

The  lists  were  presented  in  printed  test  booklets.  Each  page  of  a 
booklet  included  the  lists  to  be  compared,  the  question  stating  the 


62 


UNCLASSIFIED 


UNCLASSIFIED 


criteria  for  comparison,  and  an  answer  blank  in  which  the  code  number  of 
the  selected  list  could  be  written.  The  arrangement  of  the  lists  on 
each  page  was  varied  as  was  the  order  in  which  the  pages  were  arranged 
in  the  booklets.  Booklets  were  passed  out  to  Ss  in  random  order.  These 
controls  were  introduced  to  prevent  any  confounding  effects  which  might 
arise  from  the  order  in  which  the  lists  were  presented  and  fatigue  or 
boredom  which  might  occur  toward  the  end  of  the  session. 

4 .  Procedure 

The  70  subjects  were  told  that  the  lists  were  written  by  their  peers 
in  an  attempt  to  describe  their  actual  and  idealized  self-images  (concepts 
known  to  the  class) .  Half  of  the  subjects  were  instructed  to  compare 
lists  I  and  III  and  to  select  the  one  which  best  described  their  idealized 
self-images  and  to  compare  lists  II  and  IV  on  the  same  basis.  The  same 
two  pairings  of  lists  were  presented  to  the  other  half  with  instructions 
to  compare  them  on  the  basis  of  their  actual  self-images.  The  subjects 
were  also  presented  with  two  negative  messages  and  instructed  to  select 
the  one  which  they  and  their  peers  would  be  most  likely  to  disapprove. 
Finally,  individual  negative  category  titles  ware  listed  and  the  subjects 
indicated  the  persons  (sources)  who  would  be  most  displeased  if  they  did 
the  things  listed. 

5 .  Evaluation  of  Validity  Test  Session 

1.  The  validity  test  appeared  to  be  a  logical  method  for  evaluat¬ 
ing  the  effectiveness  of  the  data  collection  instrument.  The 
procedure  was  adequate  in  that  the  subjects  were  able  to  dis¬ 
criminate  between  the  various  lists  and  to  follow  the  instruc¬ 
tions  in  making  their  comparisons. 

2.  The  use  of  a  printed  test  booklet  was  an  efficient  method  for 
presenting  the  lists  to  be  compared  and  for  recording  the 
subjects'  responses. 
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On  the  basis  of  the  success  of  the  procedure  and  the  inherent 
logic  of  the  validity  test  design,  similar  tests  were  employed 
at  Carnation  and  Prudential. 

G.  VALIDITY  TESTS — CARNATION  AND  PRUDENTIAL 

1 •  List  Construction 

Three  methods  of  selecting  the  categories  to  be  included  in  the 
messages  were  tested  in  the  Carnation  and  Prudential  message  sessions 
which  were  held  in  the  same  week: 

1.  Categories  which  the  classifiers  ranked  as  most  important. 

Categories  containing  the  responses  which  the  subjects 
ranked  as  most  important. 

3.  Categories  that  had  the  highest  frequencies  (i.e.,  the 
greatest  number  of  responses  within  them.) 

Ii  addition,  "experts",  persons  who  had  intimate  and  detailed 
knowledge  of  the  subject  culture  but  were  not  part  of  it,  were  asked 
to  list  the  attitudes  which  they  felt  were  most  prevalent  within  and 
important  to  the  subject  population.  These  lists  were  similar  in  form 
to  the  ECHO-generated  messages  and  were  termed  "expert  messages."  The 

"experts"  employed  in  this  study  were  a  member  c£  the  personnel  depart¬ 
ment  and  a  supervising  secretary. 


The  standard  form  of  the  list  comparison  question  was  as  follows: 
"Which  of  the  messages  lists  the  actions  and  attitudes  which  you  feel 
are  most  important?" 


Message  Session  Method 


a.  Subjects 

Ninety-eight  Prudential  employees  participated  in  the  first  message 
session.  In  the  Carnation  message  session,  15  Ss  were  secretaries  and 
16  held  non-secretarial  positions. 
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b.  Standard  Procedure 

The  messages  were  printed  on  sheets  which  were  organized  into  test 
booklets.  The  standard  sheet  had  printed  on  it  the  messages  to  be  com¬ 
pared  (two  ot  four  messages),  the  question  stating  the  criteria  for  com¬ 
parison  when  different  questions  were  to  be  answered,  and  an  answer  blank. 
The  order  in  which  the  messages  were  arranged  on  any  one  sheet  was  varied 
to  control  for  possible  ordering  effects.  Likewise,  the  order  in  which 
the  sheets  were  arranged  in  the  test  booklet  was  v;.*ied  to  control  for 
effects  of  ordering,  fatigue  and  boredom. 

The  test  booklets  were  distributed  randomly  among  the  subjects  who 
were  instructed  to  read  the  messages  and  answer  the  question(s)  by  writ¬ 
ing  the  code  number  of  the  selected  message  in  the  answer  blank.  The 
subjects  were  permitted  to  >rk  at  their  own  speed  and  to  leave  the  room 
upon  completion  of  the  test. 

Special  test  booklets  were  prepared  for  different  subgroups.  For 
example,  the  Carnation  subjects  were  divided  into  two  groups,  secretaries 
and  non-secretaries,  and  different  messages,  based  on  secretarial  and 
non-secretarial  data  respectively,  were  prepared  for  them. 

c.  Variations  in  Message  Session  Procedure 

To  examine  the  effectiveness  of  the  ECHO  instrument  in  discovering 
the  power-structure,  or  hierarchy  of  sources  of  reinforcement,  the  message 
session  subjects  were  presented  with  individual  category  titles  and 
instructed  to  indicate  the  person  who  would  be  most  likely  to  approve  or 
disapprove  of  each  action  or  attitude.  The  resultant  distribution  of 
the  sources  of  approval  and  disapproval  attributed  to  a  specific  category 
was  .Later  compared  with  the  original  source  distribution. 

An  attempt  was  made  in  the  Carnation  and  Prudential  message  sessions 
to  test  the  efficacy  of  creating  a  hierarchy  of  values  on  the  basis  of 
category  frequencies  alone.  The  subjects  were  instructed  to  select 
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individual  category  titles  in  a  paired  comparison  format.  The  frequencies 
of  selection  were  compared  with  the  original  ranking  of  the  categories 
by  frequency. 

4.  Results  and  Evaluati'n 

a.  Statement  of  Criteria  for  List  Comparison 

One  result  of  the  Carnation  and  Prudential  validity  tests  was  the 
discovery  of  the  need  to  define  the  role  of  the  subject  and  to  clearly 
indicate  the  person  whose  opinion  is  being  sought. 

The  exact  wording  of  the  list  comparison  question  greatly  influenced 
the  manner  in  which  the  subjects  responded.  For  example,  when  asked  to 
select  the  "most  important"  messages,  the  Carnation  subjects  were  confused 
about  the  role  they  should  assume.  That  is,  one  message  might  be  more 
important  to  a  subject  in  terms  of  her  role  as  an  employee  while  another 
might  be  more  important  to  her  in  her  role  as  a  mother.  It  was  necessary, 
therefore,  to  define  "importance"  in  terms  of  the  role  the  subject  was  to 
assume  in  making  her  selection. 

The  following  are  examples  of  ways  in  which  the  message  comparison 
question  was  stated: 

Which  of  the  messages  described  actions  and  attitudes  which 
you  feel  are  most  important  to  doing  your  job  in  the  Company? 

Though  this  question  was  effective  in  assigning  the  employee  role,  another 
problem  arose.  The  subjects  asked:  "Important  to  whom?  To  me?  To  my  boss?" 
It  was  necessary,  therefore,  to  indicate  not  only  the  role,  but  also  the 
person  whose  opinion  was  sought. 

Which  of  the  messages  describes  the  actions  and  attitudes  in 
doing  your  job  which  are  most  important  to  your  immediate 
superiors  (i.e.,  your  boss,  supervisor  and  manager)?  That 
is,  which  message  lists  the  actions  which  they  would  be  most 
likely  to  approve  of  your  doing? 
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This  question  not  only  defined  the  subject's  role  as  an  employee,  but 

it  also  identified  the  person  whose  opinion  of  importance  was  being 
sought. 

Which  message  lists  the  actions  and  attitudes  which  you 
believe  are  the  most  important  aspects  of  doing  your  job  and 
to  being  satisfied  and  personally  content  in  your  work? 

This  question  was  effective  in  assigning  the  subject's  role  and  in 
eliciting  his  personal  opinion. 

A  second  validity  test  was  held  at  Prudential  to  reevaluate  the 
new  message  selection  criteria.  When  the  roles  were  clearly  defined, 
the  subjects  chose  the  ECHO-generated  frequency  messages.  These  findings 
were  applied  in  all  subsequent  sessions. 

b.  Evaluation  of  List  Construction 
(1)  Importance  Rankings 

The  subject's  rankings  of  their  responses  in  terms  of  importance 
were  found  to  be  ineffective  as  a  meant:  of  ranking  the  attitudes  held 
by  the  whole  group  since  the  individual  importance  ranks  were  randomly 
distributed  throughout  the  categories  (see  Appendix  II) . 

This  study,  while  not  conclusive,  indicated  that  the  importance 
rankings  assigned  to  the  categories  by  the  classifiers  were  not  represen¬ 
tative  of  the  hierarchy  of  attitudes  held  by  the  subject  population. 

This  conclusion  was  reached  when  the  subjects  regularly  rejected  the 
messages  created  on  the  basis  of  classifiers'  rankings. 

(2)  Category  Frequencies 

Selection  of  those  categories  containing  the  highest  frequencies 
appeared  to  be  the  most  effective  means  to  date  for  discovering  some 
prevalent  attitudes  within  the  value  system  of  the  subject  population. 
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H.  DEVELOPMENT  OF  CROSS-CLASSIFICATION  PROCEDURE— UCLA 

I.  Purpose 

The  cross-classification  or  triangulation  procedure  was  designed  to 
accomplish  two  goals:  (1)  To  objectively  redefine  the  meanings  of 
category  labels  created  by  the  classifiers;  (2)  to  quantify  differences 
in  social  perception. 

The  end  result  of  a  classification  session  was  a  system  of  category 
titles  which  defined  the  individual  responses  within  the  categories. 

These  titles  were  limited  as  they  represented  the  meanings  attributed  to 
the  responses  by  only  one  team  of  classifiers.  In  the  triangulation  pro¬ 
cedure,  at  least  three  different  teams  classified  the  same  data.  In  this 
manner,  the  meaning  of  a  category  title  was  redefined  in  terms  of  the 
category  labels  assigned  by  the  other  classification  teams  to  the  respon¬ 
ses  which  were  included  in  the  first  category.  These  different  labels 
represented  different  ways  of  verbalizing  the  content  of  the  responses 
and  served  to  make  the  description  of  those  responses  much  more  detailed 
than  the  category  title  resulting  from  one  classification  system. 

For  example,  the  sample  was  classified  by  classification  teams 
M2’  anc*  F4’  T^e  highest  ranking  category  (in  terms  of  frequency) 
created  in  the  M3  classification  was  Category  52,  "Be  a  good  student." 

The  category  labels  assigned  by  the  M2  Cs  to  the  responses  in  M3's  category 
52  included:  "Get  good  grades  and  do  well  in  school",  "Be  accepted  and 
make  good  in  graduate  school",  "Improve  my  personal  and  physical  being", 

"Be  a  better  person  and  relate  to  my  environment  better",  and  "Find  a 
satisfying  occupation".  Likewise,  the  category  labels  assigned  to  the 
same  responses  by  the  F^  Cs  included:  "Do  better  in  school  because  of 
outside  pressure",  "Go  to  some  sort  of  graduate  school",  "Improve  sphere 
of  knowledge",  "Realize  minor  goals",  "Graduate  within  allotted  time", 

"Have  more  self-confidence",  and  "Be  more  livable".  It  can  be  seen  that 
cross-classification  leads  to  a  more  detailed  and  specific  definition  of 
the  meaning  of  Mj’s  category  label  "Be  a  good  student". 
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To  quantify  differences  in  social  perception,  the  compositions  of 
the  different  teams  classifying  the  data  were  varied.  For  example,  a 
team  of  men  and  a  team  of  women,  when  classifying  the  same  responses, 
created  different  categories  presumably  due  to  the  differences  in  their 
interpretations  of  the  data.  Cross— classification  is  a  means,  then,  for 
discovering  differences  in  social  perception  between  different  subcul¬ 
tures  or  groups  within  the  subject  population. 

2.  Data  Base 

The  data  samples  were  chosen  randomly  from  the  positive  responses 
made  by  the  subjects  in  the  UCLA  data  collection  session.  The  responses 
made  by  15,  15,  and  22  female  subjects  constituted  three  female  samples, 

T?2 »  and  F3»  respectively.  The  three  samples  of  male  positive  respon¬ 
ses  (Mp  M2,  and  M3)  were  generated  by  15,  15,  and  30  male  subjects, 
respectively. 

In  addition,  the  M1  and  M2  samples  were  combined  as  were  the  Fx  and 
F2  samples.  These  combinations  resulted  in  two  "new"  samples  of  30 
subjects  each  which  were  also  classified. 

The  samples  containing  15  subjects  each  were  considered  sufficiently 
large  on  the  basis  of  the  Stanford  sample  size  experiments.  In  addition, 
each  of  these  samples  consisted  of  approximately  150  cards,  small  enough 
to  prevent  the  confounding  effects  of  fatigue. 

3.  Procedure 

a.  Composition  of  Classification  Teams 

Five  all-male  classification  teams,  each  one  composed  of  three 
classifiers  (teams  M2>  M3,  ,  and  M5>  and  four  all-female  teams  of 

three  classifiers  each  (teams  F^ ,  F2>  F3,  and  F^)  were  employed  in  the 
classification  of  the  samples  drawn  from  the  UCLA  positive  data.  Table 
10  indicates  which  samples  were  claseified  by  each  of  the  classification 
teams . 
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TABLE  10 

CLASSIFICATION  OF  UCLA  SAMPLES 


An  "X"  indicates  that  the  team  (in  a  specific  row)  classified  the 
sample  (in  a  specific  column) . 

Classification 

Team  M^  M2  M3  Fx 

X 

XX  X 

X  XX 


X  X 

X  XX 

XX  X 

X 


Each  of  the  classification  teams  followed  the  standard  classifica¬ 
tion  procedure:  initial  (individual)  sorting  of  cards;  formation  of  teams 
and  creation  of  larger  categories;  refinement  of  categories  and  assignment 
of  labels.  The  Cs  did  not  rank  the  categories. 

4.  Resul ts 

a.  Time  Factors 

The  length  of  time  taken  to  complete  any  one  classification  is  affec¬ 
ted  by  the  size  of  the  task  (i.e.,  the  number  of  cards  to  be  classified), 


M, 


M„ 


M„ 


M, 


M, 


b. 


Method 


Sample 

F2  F3  and  M^)  (F^  and 
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the  length  of  time  necessary  for  the  team  to  learn  the  classification 
procedure,  monotony,  boredom,  fatigue,  motivation,  and  skill. 

b.  Optimal  Conditions 

•  The  classification  sessions  should  not  exceed  three  hours  in 

length,  and  there  should  be  substantial  rest  periods  between 
sessions. 

•  IWo  hundred  fifty  to  300  cnrdo  Is  the  optimal  number  of  cards 
that  can  be  handled  efficiently  by  one  team  of  three  classi- 
fiers  in  one  session. 

I .  CUBAN  STUDIES 

1.  Purpose 

The  application  of  the  ECHO  methodology  to  a  somewhat  alien  culture 
in  a  language  other  than  Engli:  ,as  seen  as  a  logical  step  in  the  process 

of  developing  the  ECHO  instrument  for  eventual  use  with  a  completely  for¬ 
eign  culture. 


2.  Subjects 

T»o  groups  of  Cuban  refugees  living  In  the  United  States  served  as 
subjects.  Ihe  first  group.  CSl,  was  composed  of  11  women  and  14  men- 
the  second  group.  K,  included  10  men  and  9  women.  All  the  subjects,  whose 
ages  ranged  from  17  to  69  years,  were  literate  and  most  of  them  had  at 
least  a  high  school  education.  The  two  groups  will  be  considered  together, 
as  both  received  the  same  experimental  treatment  and  conditions. 


^ *  Data  Collection  Session 


a.  Materials 

Data  collection  packets  were  in  standard  form;  all  the  materials 
were  printed  in  Spanish.  As  with  previous  groups,  the  presentation  order 
of  the  positive  and  negative  cards  was  varied.  The  subjects  received 
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biographical  information  cards  which  requested  information  about  sex,  age, 
marital  status,  education,  occupation,  and  length  of  residence  in  the 
United  States. 

b.  Procedure 

A  man  indigenous  to  the  subject  group  read  the  instructions  in  Spanish 
to  the  subjects.  The  standard  data  collection  procedure  was  followed  in 
the  distribution  and  completion  of  the  packets. 

4.  Data  Classification  Session 

The  standard  classification  procedure  was  employed.  The  several 
variations  in  the  composition  of  classification  teams  and  the  samples 
of  data  classified  are  shown  in  Table  11. 

5.  Validity  Test 

a.  Construction  of  Lists 

Several  "Frequency"  messages  were  constructed  based  on  the  classifi¬ 
cation  systems  shown  in  Table  11.  In  addition,  the  "Frequency"  message 
from  the  Stanford  positive  data  was  included. 

b.  Procedure 

Pencil  and  paper  test  booklets  were  prepared,  each  page  containing 
two  lists  printed  in  Spanish  and  an  answer  blank.  The  subjects  were 
instructed  to  select  from  each  pair  of  lists  "the  one  which  most  exactly 
describes  the  things  which  you  believe  are  most  important  to  you".  The 
booklets  received  by  12  male  subjects  contained  message  pairings  which 
differed  slightly  from  those  prepared  for  the  13  females. 

6.  Results  and  Implications 

a.  Data  Collection  Session 

The  Cuban  study  represented  the  first  application  of  the  ECHO  metho¬ 
dology  with  persons  speaking  a  language  other  than  English.  After  the 
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data  collection  questions  had  been  translated  into  Spanish,  no  problems 
arose  from  the  use  of  a  foreign  language,  as  it  was  not  necessary  to 
translate  the  responses  or  category  titles  into  English.  All  statistical 
and  computer  manipulation  of  the  data  utilized  the  code  numbers  assigned 
to  the  categories,  and  in  this  way,  the  meaning  or  context  of  the  category 
labels  created  by  the  indigenous  classifiers  was  unaltered. 

b.  Classification  Session 

The  cross-classifications,  e.g.,  the  classification  of  male  response 
by  female  classifiers,  revealed  attltudinal  differences  between  males  and 
females  in  the  subject  culture.  Although  the  females  were  often  astounded 
by  and  unable  to  classify  some  of  the  male  responses,  they  were  able  to 
accept  those  responses  as  normal  and  understandable  when  they  were  presen¬ 
ted  in  the  context  of  the  classification  system  created  by  the  males, 

i.e.,  they  were  given  the  cards,  with  the  male  category  labels  attached, 

and  were  asked  if  the  classifications  were  appropriate. 

c.  Validity  Test 

As  a  result  of  the  validity  test,  some  evidence  was  found  regarding 
the  question  of  whether  positive  and  negative  responses  should  be  classi¬ 
fied  together  (i.e.,  mixed  prior  to  the  classification  session)  or  separ¬ 
ately.  The  Cuban  males  were  presented  with  two  lists:  Message  I,  male 
positive  data  classified  by  males,  ar.d  Message  V,  combined  male  positive 
and  negative  data  classified  by  males.  Likewise,  the  female  subjects 
were  instructed  to  compare:  Message  III,  female  positive  data  classified 
by  females,  and  Message  VI,  combined  female  positive  and  negative  data 
classified  by  females.  In  selecting  the  most  important  list,  both  groups 
selected  the  list  based  upon  the  separate  classification  of  positive 
responses.  That  is,  they  rejected  the  lists  which  were  generated  from 
the  classification  of  combined  positive  and  negative  responses;  the 
results  were  significant  beyond  the  0.10  a  level. 
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J.  OPERATIONAL  EVALUATION  OF  ECHO  METHODOLOGY— NORTHWESTERN  UNIVERSITY 
AND  NURSES 

1.  Purpose 

Two  new  subject  populations  were  tested  as  an  operational  evaluation 
of  the  ECHO  methodology  which  had  been  developed  and  refined  through 
previous  applications. 

2.  Subleccs 

The  two  samples  consisted  of  68  students  enrolled  in  an  undergrad¬ 
uate  psychology  class  at  Northwestern  University  and  52  female  nursing 
students.  Both  groups  of  subjects  were  predominantly  freshmen  and  sopho¬ 
mores  and  ranged  from  18  to  22  years  old. 

3.  Data  Collection  Session 

a.  Materials 

Data  collection  packets  were  prepared,  each  consisting  of  10  IBM 

cards  printed  with  the  questions  "What  is  a  good  thing  to  do?  Who  would 
approve?  and  10  cards  printed  with  the  questions  "What  is  a  bad  thing 
tc  do?  Who  would  disapprove?"  A  biographical  information  card,  placed 
last  in  the  packet,  solicited  information  regarding  the  subject's  age, 
sex,  marital  status,  and  academic  status,  and  the  number  of  years  he  had 
attended  the  school  at  which  he  was  then  enrolled.  In  addition,  the 
subject  was  instructed  to  write  on  the  biographical  information  card  the 
person  whose  approval  and  disapproval  was  most  important  to  him. 


The  Northwestern  subjects  were  randomly  divided  into  two  equal 
groups  (Gr;.up  R  and  Group  S)  and  the  Nurses  into  two  groups  (Group  P  and 
Group  Q).  These  divisions  were  made  merely  to  decrease  the  sizes  of  the 

samples,  and,  hence,  the  number  of  responses  to  be  classified  by  each 
team. 
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b.  Procedure 

The  packets  were  distributed  among  the  subjects  who  were  instructed 
to  work  at  their  own  speed  in  answering  the  questions  printed  on  the  cards. 
Importance  rankings  of  the  responses  were  omitted  as  they  were  found,  in 
the  Carnation  and  Prudential  studies,  to  be  random  and  unreliable  esti¬ 
mates  of  the  importance  hierarchy  of  attitudes. 

4.  Data  Classification 


a.  Classifiers 

Seven  teams  of  classifiers  were  chosen  randomly  from  the  Northwestern 
subject  population.  The  composition  of  the  teams  was  as  fellows; 


Team  Number 
1 
2 

3 

4 

5 

6 

8  (sic) 


Description 
Two  males ,  one  female 
Two  males,  one  female 
One  male,  two  females 
Two  males,  one  female 
Two  males,  one  female 
Three  females 
Three  females 


Eight  classification  teams,  each  consisting  of  three  women,  were 
chosen  randomly  from  the  Nurses  subject  population. 

b.  Cross-Classification — Distribution  of  Samples 

The  data  collection  sessions  resulted  in  four  Northwestern  samples 
(R+  and  S+,  positive  responses,  and  R-  and  S-,  negative  responses)  and 
four  Nurses  samples  (P+  and  Q+,  positive  responses,  and  P-  and  Q-,  nega¬ 
tive  responses) .  Tables  12  and  13  indicate  which  samples  were  classified 
by  each  of  the  classification  teams. 
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TABLE  12 

CLASSIFICATION  OF  NORTHWESTERN  SAMPLES 

An  "X"  indicates  that  the  team  (in  a  specific  row)  classified  the 
sample  (in  a  specific  column).  "N"  equals  the  total  number  of  cards  in 


each  sample. 

Classification 

Team 

R+(N  -  330) 

R-(N  - 

Sample 

325)  S+(N 

-  347) 

S-(N  -  340) 

1 

X 

X 

2 

X 

X 

3 

X 

X 

4 

X 

X 

5 

X 

6 

X 

8  (sic) 

X 

c.  Procedure 

Standard  data  classification  procedure  was  employed  with  both 
groups.  However,  a  reliability  estimate  was  Introduced  in  the  Nurses 
classification.  A  third  team  of  classifiers  was  given  the  cards  in  each 
sample  and  instructed  to  sort  them  into  the  categories  previously  created 
by  another  team. 

5.  Validity  Tests 

a.  Construction  of  Lists 

Frequency  messages  (lists)  were  based  on  each  of  the  classification 
systems.  In  addition,  frequency  messages  were  constructed  from  Stanford, 
Cuban,  and  UCLA  data.  Finally,  two  sex-specific  messages  were  based  on 
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the  separate  distributions  of  the  male  and  female  responses  among  the 
categories  created  by  team  6. 

b.  Procedure 

The  test  booklets  consisted  of  various  combinations  of  the  messages, 
presented  two  per  page.  From  each  pair  of  lists,  the  Ss  were  instructed 
to  select  the  one  list  which,  in  their  opinion,  was  more  important  overall. 
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In  order  to  present  different  combinations  of  lists,  the  Northwest¬ 
ern  subjects  were  divided  into  three  male  and  three  female  groups  and 
different  booklets  were  prepared  for  them.  Likewise,  two  different  book¬ 
lets  were  prepared  for  the  Nurses  subjects.  The  booklets  were  distributed 
and  the  subjects  instructed  to  complete  them  at  their  own  speed. 

6.  Results 

The  results  were  encouraging  and  interesting,  as  the  validity  test 
Ss  were  able  to  discriminate  between  lists  of  attitudes  on  the  basis  of 
the  sex  and  school  of  the  data  collection  subjects  from  whose  responses 
the  lists  were  generated. 

In  addition,  the  applications  of  the  ECHO  methodology  to  the  Nurses 
and  the  Northwestern  students  reconfirmed  the  logic,  and  efficiency  of 
the  procedure  itself.  The  method  as  tested  was  the  culmination  of  the 
various  development  changes  which  resulted  from  previous  studies.  As  of 
this  writing,  additional  data  are  being  collected  from  these  populations. 

K.  ELEMENTARY  SCHOOL  CHILDREN 

Data  were  collected  from  two  groups  of  school  children  (3rd  and  6th 
grades)  to  answer  two  specific  questions.  First,  can  the  IBM  card  form 
of  the  ECHO  instrument  be  applied  to  children,  and  what  is  the  minimum 
age  for  Ss  using  this  form?  And  secondly,  can  the  ECHO  technique  detect 
and  describe  the  process  of  "value  internalization,"  i.e.,  the  socializa¬ 
tion  process  and  the  sources  of  reinforcement  which  determine  its  course? 
The  following  tentative  findings  are  indicated: 

1.  Third  Graders 

"Above  average"  school  children  (ages  8-8  1/2) ,  in  the  last  quarter 
of  the  third  grade  are  capable  of  handling  the  IBM  card  form  of  the  ECHO 
questionnaire.  The  Ss  understood  the  questions,  were  easily  motivated, 
and  were  very  conscientious  in  completing  the  questionnaire.  Children, 
in  fact,  seem  to  be  more  motivated  than  adult  Sis — every  child  completed 
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all  20  questions;  adults  sometimes  fail  to  finish  the  20  questions.  A 
drawback,  however,  is  that  third  graders  require  1  hour  to  1  hour  and  20 
minutes  to  finish  the  questionnaire,  while  the  average  adult  requires 
only  20  minutes  to  complete  20  questions  and  rank  the  cards  on  an  "impor¬ 
tance  hierarchy". 

Third  grade  Sis  who  were  identified  as  being  "average"  or  "below 
average"  by  the  teacher  were  not  able  to  handle  the  IBM  form  of  the  instru¬ 
ment  in  an  acceptable  time  period.  These  same  Ss ,  however,  were  able  to 
answer  the  20  ECHO  questions  when  the  task  required  answers  to  be  listed 
on  an  8  1/2"  x  11"  piece  oi  lined  paper.  These  findings  suggest  that  all 
"normal"  American  fourth  graders  could  easily  handle  the  IBM  card  instrument. 

2.  Sixth  Graders 

Sixth  graders  (ages  11-12)  were  tested  with  the  standard  20  question 
ECHO  instrument.  The  data  were  compared  with  data  collected  from  third 
graders  in  an  effort  to  identify  and  quantify  the  changes  in  values  which 
occur  during  the  3-year  period  between  3rd  and  6th  grade.  This  experiment 
indicates : 

1.  The  ECHO  instrument  can  detect  shifts  in  the  prevalence  and 
intensity  of  values  in  a  culture. 

2.  The  method  can  also  identify  the  "power  structure",  i.e., 
sources  of  reinforcement,  which  shaped  these  changes.  In 
effect,  this  tells  the  researcher  what  (or  who)  influenced 
each  J3s'  evaluation  of  the  person,  object,  or  concept  under 
study. 

3.  Changes  in  the  "power  structure"  can  also  be  used  to  detect 
the  "internalization"  process,  i.e.,  the  process  of  adopting 
the  beliefs,  norms,  and  values  of  one's  referent  culture. 
Techniques  for  representing  this  information  quantitatively 
and  graphically  have  also  been  developed. 
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4.  TaKen  together,  findings  (1)  and  (2)  indicate  that  "develop¬ 
mental"  investigations  can  be  executed  by  the  ECHO  technique 
within  a  relatively  short  period  of  time,  i.e.,  it  is  not 
necessary  to  wait  for  time  to  pass  to  study  changes  in  values 
if  it  can  be  inferred  that  the  differences  between  two  age 

groups  of  children  in  the  same  culture  are  due  to  the  culture's 
socialization  process. 

L.  ILLITERATE  SUBJECTS 

A  pilot  study  designed  to  evaluate  the  feasibility  of  an  oral  inter¬ 
view  form  of  the  ECHO  questionnaire  is  currently  in  progress. 

1.  Subjects 

Mexican-Americans  living  in  the  "East  L.A."  area  of  Los  Angeles  acted 
as  subjects.  Each  subject  was  interviewed  by  two  researchers;  responses 
were  recorded  on  IBM  cards.  Use  of  this  particular  population  permits  the 
continued  evaluation  of  the  instrument's  ability  to  work  in  an  "alien 
culture"  and  at  the  same  time  provides  information  about  the  problems 
inherent  in  testing  illiterates. 

2.  Findings 

Potential  problems  in  applying  the  ECHO  technique  to  illiterates 
include: 

1*  Loss  of  1  anonymity— this  is  particularly  confounding  in 
societies  where  extreme  ingratiation  is  a  cultural  trait. 

2.  Excessive  time  requirements— these  Ss  required  one  to  two 
hours  to  answer  20  questions  verbally;  literate  Ss  can  answer 
20  written  questions  in  twenty  minutes. 

3.  A  method  of  classifying  the  data  which  utilizes  indigenous 
classifiers  must  be  developed. 

4.  Validity  tests  (message  tests)  for  illiterate  populations  must 
be  developed.  Tape  recorded  lists,  presented  with  rigid 
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experimental  controls  for  experimenter  bias,  presentation 
effect,  etc.,  are  under  consideration  as  a  possible  solution 
to  the  problem. 

Possible  solutions  to  these  and  other  problems  have  been  given 
some  consideration  during  Phase  I;  however,  empirical  evidence  has  not  yet 
been  collected. 
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APPENDIX  II 

CONSTRUCTION  OF  VALIDITY-TEST  MESSAGES 

The  format  of  the  validity  tests  requires  that  data  from  the  target 
population  and  data  from  several  other  populations  be  "echoed"  back  to  a 
sample  of  the  target  population  for  approval  rr  disapproval.  The  hypothe¬ 
sis  is  that  subjects  will  select  the  data  that  came  from  their  population 
over  data  from  any  other  population.  The  data  in  this  case  are  category 
titles  which  were  generated  by  teams  of  classifiers.  The  classification 
process  produces  lists  of  category  descriptions,  frequencies  of  mention 
for  each  category,  and  classifiers'  importance  rankings  of  each  category. 
Other  information  is  available  within  each  category:  the  importance 
ranking  each  J3  gave  to  the  cards  he  submitted,  the  order  in  which  the 
cards  in  a  category  are  completed,  the  number  of  Ss  who  contributed  to 
the  category,  and  how  other  classifiers  sorted  the  same  cards. 

Problem.  What  portion  of  the  available  data  contains  the  most 
veridical  information  about  the  subject  population's  value  system,  i.e., 
which  of  the  possible  samples  of  the  data  will  they  say  is  most  important 
to  them? 

Method.  Generate  lists  of  values  in  different  ways  and  see  which 
method  produces  preferred  lists.  Lists  were  produced  in  the  following 
ways: 

1.  Experts  submitted  lists  of  values  which  they  thought  the 
population  held.  The  first  five  items  were  included  in  the 
"Expert's"  list. 

2.  Classifiers  were  asked  to  rank  all  of  their  categories  on  an 
importance  scale.  The  five  highest-ranked  categories  were 
selected  for  the  "importance  to  Classifiers"  list. 


UNCLASSIFIED 


8:3 


UNCLASSIFIED 


3.  The  five  categories?  with  the  highest  frequency,  i.e.,  the 
categories  with  the  largest  number  of  cards  in  them,  were 
included  in  uhe  "Frequency"  list. 

4.  Each  subject  had  ranked  his  responses  on  an  importance  scale; 
by  taking  the  responses  ranked  "1"  it  was  hoped  that  a  weight¬ 
ing  for  importance  could  be  achieved.  The  five  categories 
with  the  largest  number  of  "1"  importance  values  were  included 
in  the  "Importance  to  Ss"  list. 

The  lists  were  presented  in  a  paired-comparison,  forced-choice  format  and 
Ss  were  asked  to  select  the  list  which  best  reflected  their  views. 

Results.  The  "Frequency"  lists,  (3)  above,  proved  to  be  the  most 
popular  of  the  lists  in  several  different  validity  test  sessions. 

Discussion.  "Importance  to  Ss"  rankings  and  categories  proved  to 
be  statistical lv  independent,  i.e.,  no  category  came  up  with  a  prepon¬ 
derance  of  the  higher  "importance"  ratings  (see  Table  14  for  example). 

By  taking  the  distribution  of  all  of  the  cards  ranked  "1"  we  were  merely 
sampling  10%  of  the  population  data  distribution.  The  Law  of  Large 
Numbers,  which  is  applicable  here,  tells  us  that  the  variance  increases 
as  the  sample  size  decreases.  By  taking  1/10  of  the  responses  we  merely 
increased  the  variance  in  the  frequency  distribution  from  which  we  selec¬ 
ted  the  five  largest  categories.  This  method  of  selecting  categories 
would  not  be  expected  to  produce  "preferred"  lists;  and,  in  fact,  it  did 
not. 


Another  interesting  characteristic  of  the  "importance  ranking  wc*s 
discovered  in  the  course  of  post-test  data  analysis  importance  rankings 
are  dependently  related  to  the  order  in  which  the  cards  were  filled  out. 
That  is,  the  cards  filled  out  first  were  usually  given  the  higner  (I) 
rankings.  Table  15  illustrates  the  dependency  finding  which  obtained 
in  all  data  checked. 
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TABLE  14 

DISTRIBUTION  OF  SUBJECTS'  IMPORTANCE  RANKINGS  BY  CATEGORY 
(Stanford  Positive  Data) 
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TABLE  15 

CORRELATION  BETWEEN  ORDER  AND  IMPORTANCE  OF  ITEMS 
Response  Sequence  Number 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

1 

18 

6 

12 

5 

9 

5 

1 

4 

4 

8 

72 

2 

13 

18 

6 

4 

7 

3 

3 

9 

4 

4 

71 
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7 

16 

7 

11 

7 
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2 

8 

4 

4 

70 

Chi  Square  =  180.961 
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10 

2 

4 

4 

7 

4 
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5 

11 

8 

17 
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NA 

2 

2 

3 

2 

2 

3 

2 

2 

2 

2 

22 

73 

73 

73 

73 

73 

70 

69 

70 

70 

70 

714 

Classifier's  Rankings  of  Importance.  The  reliability  with  which 
different  teams  of  classifiers  rank  the  same  categories  in  the  same  order 
of  importance  appears  to  be  high.  For  example,  the  correlation  between 
the  importance  rankings  of  two  different  teams  of  Company  C  classifiers 
was  found  to  be  +.79;  the  data  and  statistical  tests  which  support  this 
statement  are  contained  in  Table  16. 

The  relationship  between  categories  ranked  on  importance  by  the  Cs 
and  ranked  on  frequency  of  mention  is  not  so  striking,  as  the  data  in 
Table  17  illustrate.  The  failure  of  the  "importance  to  Cs"  technique 
can  probably  be  attributed  to  Cs  variance;  the  "Frequency"  technique 
minimizes  (but  does  not  eliminate)  the  variance  between  Cs. 
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TABLE  16 

CLASSIFIERS'  IMPORTANCE  RANKINGS 
(Two  Groups  of  Secretaries  Classifying  Data 
from  the  Second  Group) 


Rank  Assigned  Rank  Assigned  Difference  >  d, 
by  First  Group  by  Second  Group  Between  Rank  No’s  d 
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13 

1 
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TABLE  17 

RELATIONSHIP  OF  CLASSIFIERS'  IMPORTANCE  RANKINGS 
OF  CATEGORIES  TO  CATEGORY  FREQUENCIES 


Cs 

Data 

Spearman  rJio 

Significance  of  Rho 

PI 

Prudential  + 

p  =  0.36 

t 

=  1.724 
df  =  20 

N.S. 

P2 

Prudential  - 

p  -  0.03 

t 

=  0.156 
df  =  27 

N.S. 

P3 

Prudential  + 

p  =0.83 

t 

=  6.308 
df  =  18 

+ 

a  =  . 001+ 

CA 

Carnation  + 

p  =  0.42 

t 

=  2.36 
df  =  26 

+ 

a  =  .05+ 

CB 

Carnation  + 

p  =  0.18 

t 

=  1.008 

N.S. 

df  -  28 


An  Important  use  of  Cs  ranH.ngs  of  categories  vill  be  as  a  check 
against  spurious  f  rankings.  Although  the  multiple  classification  method 
normally  reveals  small  categories  that  could  be  considered  to  be  the 
breakdowns  of  larger  categories,  Cs  rankings  could  be  used  as  a  check 
against  the  possibility  that  all  three  teams  used  the  same  category 
widths  but  disagreed  about  the  classifications. 
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APPENDIX  III 

ELEMENTARY  SCHOOL  STUDIES 

The  experiments  reported  here  were  designed  to  answer  two  major 
questions:  (1)  How  old  and  literate  must  our  subjects  be  to  use  the 
standard  IBM  card  form  of  the  ECHO  questionnaire?  (2)  Can  the  ECHO 
technique  detect,  and  perhaps  quantify,  the  process  of  "value  internal¬ 
ization"  or  "socialization"? 

Subjects 

Group  A:  female  and  male  students  in  the  third  grade  at  a  public 

elementary  school  in  Santa  Ana  (Orange  County),  California. 

Group  B:  female  and  male  students  from  a  sixth  grade  at  the  same 
elementary  school. 

Method.  A  standard  twenty-card  instrument  containing  the  questions: 

"What  is  a  good/bad  thing  to  do’"  and  "Who  would  approve/disapprove?"  was 
completed  by  each  S.  The  order  of  presentation  (good/bad  or  bad/good) 
was  reversed  for  half  of  each  sample  to  control  for  sequential  effect. 

Standard  instructions,  modified  for  the  younger  populations,  were 
read  to  both  groups. 

A.  AGE  AND  LITERACY  REQUIRED  TO  USE  THE  ECHO  INSTRUMENT 

~-0blem'  Hcw  0lr7  an<?  llterate  must  subjects  (Ss)  be  to  use  the  standard 
IBM  card  form  of  the  ECHO  questionnaire?  Can  children  comprehend  the 
task?  Do  children  generate  data  that  are  meaningful? 

Bavelas  and  Kalhorn2  found  that  fourth  graders  (ages  9  to  10) 
were  the  youngest  subjects  capable  of  handling  a  written  form  of  the 
questionnaire.  Third  graders  were  excluded  from  Bavelas'  studies  because 
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their  handwriting  was  illegible.  All  of  his  subjects,  including  pre¬ 
schoolers  who  were  given  oral  interviews,  were  able  to  comprehend  the 
task  and  generate  the  requested  responses. 

Findings.  Second-semester  third  grade  students  whom  the  teacher  rated 
as  "average  and  above"  were  able  to  handle  the  standard  ECHO  instrument; 
"below  average"  students,  however,  could  not  satisfactorily  handle  the 
IBM  cards  even  though  they  could  give  acceptable  responses  when  given  an 
ordinary  piece  of  writing  paper. 

These  third  grade  students  required  one  and  a  half  hours  to  com¬ 
plete  twenty  cards  as  compared  to  an  average  adult  time  of  20  minutes. 

The  sixth  graders  completed  the  task  in  approximately  the  same  time  as 
adults. 

Conclusion.  Children  with  a  fourth  grade  education  or  more  are  capable 
of  handling  the  standard  ECHO  instrument.  Third  graders  can  answer  the 
questions  but  they  require  a  long  time  to  complete  the  task.  This  find¬ 
ing  will  vary  from  culture  to  culture,  but  it  suggests  that  the  current 
ECHO  methodology  can  be  used  at  the  lower  levels  of  education. 

B.  MEASUREMENT  OF  "SOCIALIZATION" 

Problem.  As  children  mature,  they  appear  to  internalize  the  values 
of  the  subculture  in  which  they  live.  Adults  commonly  have  sets  of  valves 
which  cause  them  to  behave  in  particular  ways  because  they  view  the  be¬ 
havior  as  being  "right"  or  "wrong".  No  outside  agency  is  required  to 
reward  or  punish  such  behavior.  An  assumption  is  that  the  behavior  and 
the  values  attached  to  it  were  learned  in  such  a  way  that  the  enforcing 
agency  is  transferred  from  the  outside  (e.g.,  from  "mother")  to  the  inside 
("self").  The  internalization  process  presumably  occurs  over  time.  The 
question  of  when  that  occurs  might  be  answered  by  determining  the  differ¬ 
ence  in  responses  to  the  question  "Who  would  approve/disapprove?"  for 
various  behaviors  by  children  of  different  age  groups. 
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Hypothesis.  Children  in  the  sixth  grade,  having  progressed  farther 
in  the  socialization  process,  will  respond  to  the  question  "Who  would 
approve/disapprove?"  with  the  concept  "self"  significantly  more  often 

than  children  in  the  third  grade;  the  converse  is  true  for  the  concept 
"parent (s)". 


Results.  The  number  and  percentage  of  "myself"  responses  to  "Who 
would  approve/disapprove?"  for  each  group  is  shown  in  Table  18. 


TABLE  18 


MYSELF  AS  A  SOURCE  OF  APPROVAL  OR  DISAPPROVAL 
IN  TWO  POPULATIONS  OF  ELEMENTARY  SCHOOL  CHILDREN 


%  Time  "Myself"  was  Source 


Good  Thing 

Bad  Thing 

Mixed 

Male  Third  Graders 

8.3 

13.3 

10.8 

Female  Third  Graders 

20.1 

20.0 

20.4 

Male  Sixth  Graders 

29.4 

19.3 

24.7 

Female  Sixth  Graders 

21.3 

19.0 

20.1 

Figure  6  includes  the  data  from  Groups  A  and  B  and  from  an  adult 
group.  The  curves  indicate  clearly  that  the  socialization  process  moves 
sharply  in  the  period  between  the  third  and  sixth  grades. 
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Figure  6.  "Socialization":  The  Internalization  of  Value  Systems 
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APPENDIX  IV 

DETAILED  DESCRIPTION  OF  METHOD 

The  following  detailed  description  of  the  basic  ECHO  methodology 
as  of  June  1967  is  a  consolidation  of  material  appearing  in  earlier 
sections  of  this  report.  It  is  presented  in  this  form  for  convenience. 
It  includes  only  one  form  of  the  projective  question;  additions  and 
changes  are  expected. 

A.  DATA  COLLECTION  SESSION 

1.  Data  Collection  Packets 

a.  Data  Collection  Cards 

Data  collection  packets  were  prepared  prior  to  the  session,  each 
packet  consisting  of  20  IBM  cards  which  were  prepunched  with  the  follow¬ 
ing  information: 

1.  A  number  identifying  the  population  from  which  the 
sample  was  drawn 

2.  A  subject  number 

3.  A  sequence  number  to  indicate  the  order  in  which  the 
subject  wrote  his  responses 

4.  A  plus  ("+")  or  minus  to  indicate  whether  the 

question  on  the  card  was  "positive"  or  "negative" . 

Of  the  20  data  collection  cards  per  packet,  10  had  printed  on  them  the 
following  "positive"  questions:  "What  is  a  good  thing  to  do?"  and  "Who 
would  approve?"  Likewise,  the  10  "negative"  cards  had  the  questions: 
"What  is  a  bad  thing  to  do?"  and  "Who  would  disapprove?" 

A  sample  data  collection  card  is  shown  in  Fig.  7. 
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WHAT  IS  A  GOOD  THING  TO  DO? 


WHO  WOULD  APPROVE? 


Go  on  to  the  next  card  in  this  packet 


0 


Figure  7.  Sample  Data  Collection  Card 


b.  Sequential  Controls 

To  control  for  the  possible  confounding  effects  of  the  order  in 
which  the  positive  and  negative  cards  were  presented,  half  of  the  ob¬ 
jects  in  each  sample  received  10  positive  cards  first  and  then  10  nega¬ 
tive  cards;  vice  versa  for  the  other  subjects. 

c.  Role  Instructions 

When  subgroups  within  a  sample  were  to  be  assigned  a  specific  role, 
role  instruction  cards  were  placed  first  in  the  packets  so  that  the  sub¬ 
jects  would  encounter  them  before  they  responded  to  the  data  collection 
cards.  For  example,  120  of  the  UCLA  Ss  were  instructed  to  respond  to  the 
questions  in  terms  of  their  roles  as  students  at  UCLA  while  the  other  30 
— s  were  instructed  to  respond  in  terms  of  the  general  role  as  person  in 
society. 
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d.  Biographical  Data  Cards 

The  final  card  in  each  data  collection  packet  was  the  Biographical 
Information  card  with  questions  appropriate  to  the  population.  The  Car¬ 
nation  and  Prudential  subjects,  for  example,  were  asked  about  sex,  educa¬ 
tional  background,  age,  job  title,  and  length  of  employment  with  the 
company.  The  biographical  data  card  employed  with  the  UCLA  and  Stanford 
samples  asked  about  the  subject's  age,  sex,  marital  and  academic  status, 
and  the  length  of  time  he.  had  attended  that  school. 

2.  Data  Collection  Methodology 

The  data  collection  packets  were  distributed  randomly  to  the  sub¬ 
jects.  After  the  instructions  were  read  to  them,  the  subjects  were 
instructed  to  open  their  packets  and  begin.  They  were  allowed  to  pro¬ 
ceed  at  their  own  speed  and  were  asked,  upon  completion,  to  put  all  the 
materials  back  into  the  packets  and  to  return  the  packets  to  the  experi¬ 
menters. 


The  instructions  to  the  subjects  were  basically  the  same  for  all 
the  sample  studied.  The  format  was  as  follows: 


INSTRUCTIONS  TO  THE  SUBJECTS 


The  purpose  of  this  study  is  to  develop  methods  for 
improving  communications  between  different  cultures.  Any 
such  interculture  communication  is  necessarily  based  on  an 
understanding  of  the  value  systems  inherent  in  each  culture. 
We  are  currently  investigating  an  instrument  with  which  we 
hope  we  will  be  able  to  discover  and  examine  the  value  sys¬ 
tem  for  any  culture.  We  plan  to  test  the  effectiveness  of 
our  methodology  by  comparing  the  culture  which  has  developed 
in  this  group  with  a  somewhat  similar  culture  which  has 
developed  within  another  -?roup  of  secretaries  [or  students]. 

Each  of  you  should  nave  a  sealed  envelope  and  a  pen. 
These  are  the  only  materials  you  will  need.  You  will  be 
asked  to  give  ten  responses  to  each  of  the  following  ques¬ 
tions:  What  do  you  think  a  person,  like  yourself,  could  do 
that  would  be  a  good  thing  to  do  and  that  someone  would 
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approve  of  your  doing?  Who  would  approve  of  your  action? 
Likewise,  what  do  you  think  a  person,  like  yourself,  could 
do  that  would  be  a  bad  thing  to  do  and  that  someone  would 
disapprove  of  your  doing?  Who  would  disapprove? 

When  we  tell  you  to  begin,  you  are  to  open  the  en¬ 
velope  by  tapping  it  down  on  the  left  side  so  that  the 
cards  within  are  down  on  the  left.  Then  tear  off  the 
right  edge.  Take  out  the  deck  of  cards  which  are  num¬ 
bered  with  a  "1".  Do  not  remove  anything  else  from  the 
envelope.  Take  the  rubber  band  off  the  cards  but  be 
especially  careful  that  you  do  not  change  the  order  of 
the  cards.  Writing  or  printing  as  neatly  as  possible 
give  a  specific  example  of  a  good  or  a  bad  thing  to  do, 
according  to  the  question  which  is  printed  on  the  card. 
Write  that  answer  on  the  lines  which  have  been  provided 
following  the  question.  Then  write  the  title  or  position 
of  the  person  who  would  approve  or  disapprove  of  your 
action  on  the  lines  which  have  been  provided  following 
that  question.  When  you  have  answered  all  ten  cards  in 
deck  1,  put  them  back  into  the  envelope  and  take  out  deck 
"2".  Again,  we  ask  you  not  to  change  the  order  of  the 
cards.  Follow  the  same  procedure  as  before  in  writing 
your  answers  on  these  cards.  When  you  are  finished,  re¬ 
turn  these  cards  to  the  envelope  and  take  out  the  yellow 
"Biographical  Information"  card.  Answer  each  of  the  ques¬ 
tions  on  this  card  by  placing  an  "x"  in  the  appropriate 
box.  Again,  when  you  are  finished  return  the  card  to  the 
envelope  and  all  your  materials  to  one  of  us.  At  that 
time  you  may  leave.  We  wish  to  thank  you  all  again  for 
donating  your  time  to  help  make  this  study  a  success. 

Do  you  have  questions?  If  a  problem  should  arise 
while  you  are  working,  please  raise  your  hand.  You  may 
begin. 


B.  CLASSIFICATION  SESSION 

The  classification  session  was  devised  as  a  method  for  reducing 
the  vast  amount  of  data  (20  separate  responses  per  subject)  into  a 
usable  and  logical  form. 

Persons  indigenous  to  the  subject  population  interpret  the  individ¬ 
ual  responses  made  by  the  subjects  and  put  together  those  responses  which 
are  similar  in  content  and  meaning.  It  is  essential  that  indigenous  clas¬ 
sifiers  be  employed  in  this  task,  as  persons  alien  to  the  subject  population 
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and,  hence,  alien  to  that  group's  value  system,  would  make  different 
interpretations  of  the  data.  The  goal  of  the  classification  procedure 
is  to  reduce  the  masses  of  individual  responses  into  a  set  of  statements 
(categories)  which  describe  the  important  attitudes  held  by  the  subject 
population. 

1 .  Determination  of  Number  of  Classification  Teams  and  Length 

of  Session 

Teams  of  three  persons  each  were  chosen  randomly  from  the  subject 
population  to  serve  as  classifiers.  The  number  of  teams  depended  upon 
the  total  number  of  cards  to  be  classified.  Through  trial  and  error,  it 
was  determined  that  250-300  was  the  optimum  number  of  cards  that  could 
be  handled  by  a  group  of  three  classifiers.  In  cases  where  a  group  of 
classifiers  worked  with  more  than  300  cards,  boredom  and  fatigue  developed 
with  the  result  that  the  classifiers  became  less  critical  in  their  dis¬ 
criminations  and  tended  to  hurry  through  the  classification  task.  For 
example,  teams  from  UCLA  were  given  over  800  cards  to  sort.  The  process 
took  approximately  twelve  hours  and  the  resultant  categories  appeared  to 
the  investigator  to  be  rather  unreliable.  Whenever  large  numbers  of  cards 
are  to  be  processed,  the  classification  should  be  spread  over  a  larger 
period  of  time  by  having  work  sessions  of  no  longer  than  three  hours, 
with  a  substantial  break  between  sessions. 

2.  Procedure 

In  the  first  part  of  the  classification  session,  the  three  classi¬ 
fiers  worked  independently.  Each  was  given  about  100  cards,  i.e.,  one- 
third  of  the  deck  of  cards  assigned  to  his  team,  and  was  instructed  to 
stack  together  all  the  cards  which,  in  his  opinion,  said  or  meant  the 
same  thing. 

The  resultant  stacks  of  cards,  then,  represented  one  category  or 
type  of  response.  This  preliminary  sorting  task  took  roughly  one  hour 
to  complete. 
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In  the  second  part  of  the  session,  each  team  of  classifiers  was 
brought  together.  The  different  teams  were  physically  separated  from 
each  other  so  that  one  group  would  not  disturb  another.  The  experimen¬ 
ters  circulated  among  the  teams  of  classifiers,  being  careful  not  to  make 
any  suggestions  which  might  bias  the  classifiers’  decisions.  The  experi¬ 
menters  were  there,  for  example,  to  make  sure  that  no  one  member  dominated 
the  team's  discussions  and  to  expedite  the  process. 

This  phase  of  the  session  began  with  one  team  member  choosing  one 
of  his  stacks  and  reading  aloud  some  of  the  cards  it  contained.  The 
other  two  classifiers  then  added  to  that  stack  any  of  their  cards  which 
they  felt  had  the  same  meaning  as  those  being  read.  The  classifiers  took 
turns  reading  their  cards  aloud.  At  the  end  of  this  phase,  all  of  the 
classifiers'  separate  stacks  of  cards  had  been  combined  to  form  larger 
categories  of  response. 

In  the  rinal  phase,  the  classifiers  were  instructed  to  reexamine 
the  newly-created  categories  and,  where  necessary,  to  break  them  down 
into  finer,  more  exclusive  categories.  They  then  labeled  each  category, 
giving  it  a  title  which,  in  the  form  of  a  response  to  the  original  ques¬ 
tion,  would  describe  all  of  the  responses  within  the  category.  In 
addition,  the  Carnation  and  Prudential  classifiers  were  instructed  to 
rank  the  resultant  categories  from  1  to  10  in  terms  of  importance.  This 
ranking  was  later  omitted  from  the  classification  procedure  but  should 
be  retained  as  a  control  over  eccentric  classifications. 

Typically,  the  classification  session  lasted  about  three  hours  and 
resulted  in  sets  of  20-30  categories  of  response. 

3.  Reliability  Estimates 

As  one  method  for  checking  the  reliability  of  a  team's  classifica¬ 
tion  system,  a  second  team  of  three  classifiers  was  given  the  list  of 
categories  created  by  the  first  team  and  instructed  to  re-sort  the  cards 
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into  those  categories.  The  degree  to  which  the  cards  were  sorted  into  the 
same  categories  by  the  two  different  teams  determined  the  degree  of  relia¬ 
bility  of  the  first  team's  classification. 

Another  estimate  of  reliability  consisted  in  having  a  second  team 
of  classifiers  indicate  their  agreement  or  lack  of  agreement  with  the 
categories  created  by  the  first  team.  That  is,  the  members  of  the  second 
team  were  given  each  category  (stack  of  cards)  created  by  the  first  team 
and  asked  whether  or  not  the  cards  did  in  fact  belong  together  and  whether 
or  not  the  category  title  created  by  the  first  group  was  representative 
of  the  responses  in  that  category. 

4*  Cross-Classification  Systems 

Several  variations  in  the  types  of  data  classified  and  the  composi¬ 
tion  of  the  classification  teams  are  possible.  The  variables  include: 

1.  Data — sex  variable 

a.  Male  subjects'  responses  only 

b.  Female  subjects'  responses  only 

c.  Male  and  female  subjects'  responses  mixed 

2.  Data — age  variable 

a.  Responses  from  subjects  representing  different  age 

groups  can  be  classified  together  or  separately 

3.  Data — positive  and  negative  responses 

a.  Responses  to  the  "positive"  and  "negative"  questions 
can  be  classified  together  or  separately 

4.  Classifiers- — sex  variable 

a.  All  males 

b.  All  females 

c.  Males  and  females 
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5.  Classifiers — age  variable 

a.  Teams  of  classifiers  can  consist  of  only  persons  of 
a  specific  age  group  or  of  persons  of  different  ages 

6.  Classifiers — indigenous  or  alien 

a.  Although  indigenous  classifiers  are  essentia.1  to  the 

original  interpretati m  of  the  data,  it  would  be  inter¬ 
esting  to  discover  how  persons  alien  to  the  subject 
population  would  view  the  data.  In  this  manner,  dif¬ 
ferences  between  the  value  systems  of  the  subject 
population  and  the  alien  culture  might  be  discovered. 

Various  combinations  of  these  variables  are  possible  and  several 
have  been  used  to  date.  These  cross-classifications  reveal  importa’ t 
differences  between  suogroups  within  the  subject  population. 

5.  Classification  of  Sources 

The  power  structure  of  the  subject  population  is  represented  by 
the  sources  of  approval  and  disapproval  of  the  actions  listed  by  the 
subjects.  The  sources  can  be  grouped  or  classified  in  various  ways, 
e.g.,  in  terms  of  their  relationships  to  the  subjects  or  their  status  in 
the  organization.  Indigenous  classifiers  are  needed,  particularly  with 
foreign  populations,  to  determine  the  hierarchy  of  the  sources  and  for 
other  groupings  of  sources. 

C.  MESSAGE  SESSIONS 

The  message  session  was  designed  to  test  the  sensitivity  of  the 
ECHO  method;  to  determine  whether  the  subject's  responses  to  the  ECHO 
questions  do  in  fact  represent  attitudes  which  are  important  and  preva¬ 
lent  in  the  value  system  of  the  subject  culture.  Lists  of  the  category 
labels  which  represent  such  attitudes  were  called  "messages".  Basically, 
the  message  session  format  was  as  follows:  A  new  group  of  subjects  was 
chosen  from  the  subject  population.  These  subjects  were  asked  to  compare 
various  messages,  e.g.,  ECHO  versus  expert;  ECHO  positive  versus  ECHO 
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negative,  and  to  select  the  message  which  listed  the  actions  and  attitudes 
t:  y  considered  to  be  most  important  on  the  basis  of  certain  pre-stated 
criteria,  e.g.,  in  terms  of  their  roles  as  students  or  secretaries. 

Hence,  the  message  session  tested  the  effectiveness  of  the  ECHO  instru¬ 
ment  in  discovering  prevalent  attitudes  within  the  subject  population 
and  also  served  as  an  examination  of  ECHO's  ability  to  discriminate  be¬ 
tween  different  subgroups  within  the  subject  culture. 

1«  Message  Construction 

a.  ECHO  Frequency  Messages 

The  categories  within  each  classification  system  were  ranked  in 
terms  of  their  frequencies,  i.e.,  the  total  number  of  responses  in  each 
category.  The  categories  which  were  highest  in  rank  were  included  in 
the  message  and  their  labels  were  listed  in  order  of  decreasing  rank. 
Different  frequency  messages  were  constructed  based  on  the  various  clas¬ 
sification  systems. 

b.  Expert  Messages 

Experts,  persons  who  had  intimate  and  detailed  knowledge  of  the 
subject  culture  but  were  not  part  of  it.  were  asked  to  list  the  atti¬ 
tudes  which  they  felt  were  most  prevalent  within  or  important  to  the 
subject  population.  These  lists  were  similar  in  form  to  the  ECHO- 
generated  messages. 

2.  Criteria  for  Message  Comparison 

As  the  message  session  was  designed  to  be  a  test  of  the  data  col¬ 
lection  method,  it  was  necessary  that  the  question,  i.e.,  the  statement 
of  the  criteria  on  which  the  subjects  were  to  base  their  comparisons  of 
the  messages,  parallel  the  data  collection  question  to  which  the  original 
responses  were  made.  The  standard  form  of  the  question  was  as  follows: 
"Which  of  the  messages  lists  the  actions  and  attitudes  which  you  feel 
are  most^important?"  However,  it  was  discovered  early  in  the  study  that 
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the  exact  wording  of  that  question  greatly  influenced  the  manner  in 
which  the  subjects  responded.  Hence,  it  was  necessary  to  assign  a 
specific  role,  e.g.,  as  a  student  or  employee,  to  define  the  context 
in  which  the  subjects  were  to  make  their  comparisons  of  the  messages. 

In  addition,  the  question  itself  must  clearly  identify  the  person  whose 
opinion  is  being  sought.  The  question:  "Which  message  lists  the  actions 
and  attitudes  which  you  believe  are  the  most  important  aspects  of  doing 
your  job  and  being  satisfied  and  personally  content  in  your  work?"  was 
effective  in  assigning  the  subject's  role  and  in  eliciting  his  personal 
opinion  rather  than  that  of  his  supervisor  or  co-workers. 

3.  Message  Session  Method 

a.  Standard  Procedure 

The  messages  were  printed  on  sheets  which  were  organized  into  t-st 
booklets.  The  standard  sheet  had  printed  on  it  the  messages  to  be  com¬ 
pared  (two  cr  four  messages),  the  question  stating  the  criteria  for  com¬ 
parison  when  different  questions  were  to  be  answered,  and  space  for  the 
answer .  The  order  in  which  the  messages  were  arranged  on  any  one  sheet 
was  randomized  to  control  for  any  possible  ordering  effects.  Likewise, 
the  order  in  which  the  sheets  were  arranged  in  the  test  booklet  was  ran¬ 
domized  to  control  for  effects  of  ordering,  fatigue,  and  boredom. 

The  test  booklets  were  distributed  randomly  among  the  subjects 
who  were  instructed  to  read  the  messages  and  answer  the  question(s)  per¬ 
taining  to  them  by  writing  the  code  number  of  the  selected  message  in 
the  answer  blank.  The  subjects  were  permitted  to  work  at  their  own 
speed  and  to  leave  the  room  upon  completion  of  the  test. 

In  some  cases,  special  test  booklets  were  prepared  for  different 
subgroups.  For  example,  the  Carnation  subjects  were  divided  into  two 
groups,  secretaries  and  non-secretaries,  and  different  messages,  based 
on  secretarial  and  non-secretarial  data  respectively,  were  prepared  for 
them.  Similarly,  the  male  subjects  in  the  Cuban  and  Northwestern  groups 
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received  test  booklets  containing  messages  based  mainly  on  the  male 
responses  while  the  females  in  those  two  groups  received  messages  cre¬ 
ated  from  female  responses. 

b.  Variations  in  Message  Session  Procedure 

To  examine  the  effectiveness  of  the  ECHO  instrument  in  discovering 
the  power  structure,  or  hierarchy  of  sources  of  reinforcement,  the  mes¬ 
sage  session  subjects  were  presented  with  individual  category  titles  and 
instructed  to  indicate  the  person  who  would  be  most  likely  to  approve  or 
disapprove  of  each  action  or  attitude.  The  resultant  distribution  of 
the  sources  of  approval  and  disapproval  attributed  to  a  specific  category 
was  later  compared  with  the  original  source  distribution,  i.e.,  the 
sources  attributed  to  the  same  category  by  the  data  collection  subjects. 

An  attempt  was  made  in  the  Carnation  and  Prudential  message  sessions 
to  test  the  efficacy  of  creating  a  hierarchy  of  attitudes  on  the  basis  of 
category  frequencies  alone.  The  subjects  were  instructed  to  make  paired 
comparisons  between  individual  category  titles  and  the  frequencies  with 
which  the  categories  were  selected  were  later  compared  with  original 
ranking  of  the  categories  by  frequency. 
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